Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
KubeArmor is a cloud-native runtime security enforcement system that restricts the behavior (such as process execution, file access, and networking operations) of pods, containers, and nodes (VMs) at the system level.
KubeArmor leverages such as , , or to enforce the user-specified policies. KubeArmor generates rich alerts/telemetry events with container/pod/namespace identities by leveraging eBPF.
Here, we demonstrate how to define security policies using our example microservice (multiubuntu).
Process Execution Restriction
Block a specific executable ()
Explanation: The purpose of this policy is to block the execution of '/bin/sleep' in the containers with the 'group-1' label. For this, we define the 'group-1' label in selector -> matchLabels and the specific path ('/bin/sleep') in process -> matchPaths. Also, we put 'Block' as the action of this policy.
Verification: After applying this policy, please get into one of the containers with the 'group-1' (using "kubectl -n multiubuntu exec -it ubuntu-X-deployment-... -- bash") and run '/bin/sleep'. You will see that /bin/sleep is blocked.
Block all executables in a specific directory ()
Explanation: The purpose of this policy is to block all executables in the '/sbin' directory. Since we want to block all executables rather than a specific executable, we use matchDirectories to specify the executables in the '/sbin' directory at once.
Verification: After applying this policy, please get into the container with the 'ubuntu-1' label and run '/sbin/route' to see if this command is allowed (this command will be blocked).
Block all executables in a specific directory and its subdirectories ()
Explanation: As the extension of the previous policy, we want to block all executables in the '/usr' directory and its subdirectories (e.g., '/usr/bin', '/usr/sbin', and '/usr/local/bin'). Thus, we add 'recursive: true' to extend the scope of the policy.
Verification: After applying this policy, please get into the container with the 'ubuntu-2' label and run '/usr/bin/env' or '/usr/bin/whoami'. You will see that those commands are blocked.
Allow specific executables to access certain files only ()
Explanation: Here, we want the container with the 'ubuntu-3' label only to access certain files by specific executables. Otherwise, we want to block any other file accesses. To achieve this goal, we define the scope of this policy using matchDirectories with fromSource and use the 'Allow' action.
Verification: In this policy, we allow /bin/cat to access the files in /credentials only. After applying this policy, please get into the container with the 'ubuntu-3' label and run 'cat /credentials/password'. This command will be allowed with no errors. Now, please run 'cat /etc/hostname'. Then, this command will be blocked since /bin/cat is only allowed to access /credentials/*.
Allow a specific executable to be launched by its owner only ()
Explanation: This policy aims to allow a specific user (i.e., user1) only to launch its own executable (i.e., hello), which means that we do not want for the root user to even launch /home/user1/hello. For this, we define a security policy with matchPaths and 'ownerOnly: ture'.
Verification: For verification, we also allow several directories and files to change users (from 'root' to 'user1') in the policy. After applying this policy, please get into the container with the 'ubuntu-3' label and run '/home/user1/hello' first. This command will be blocked even though you are the 'root' user. Then, please run 'su - user1'. Now, you are the 'user1' user. Please run '/home/user1/hello' again. You will see that it works now.
File Access Restriction
Allow accessing specific files only ()
Explanation: The purpose of this policy is to allow the container with the 'ubuntu-4' label to read '/credentials/password' only (the write operation is blocked).
Verification: After applying this policy, please get into the container with the 'ubuntu-4' label and run 'cat /credentials/password'. You can see the contents in the file. Now, please run 'echo "test" >> /credentials/password'. You will see that the write operation will be blocked.
Block all file accesses in a specific directory and its subdirectories ()
Explanation: In this policy, we do not want the container with the 'ubuntu-5' label to access any files in the '/credentials' directory and its subdirectories. Thus, we use 'matchDirectories' and 'recursive: true' to define all files in the '/credentials' directory and its subdirectories.
Verification: After applying this policy, please get into the container with the 'ubuntu-5' label and run 'cat /secret.txt'. You will see the contents of /secret.txt. Then, please run 'cat /credentials/password'. This command will be blocked due to the security policy.
Network Operation Restriction
Audit ICMP packets ()
Explanation: We want to audit sending ICMP packets from the containers with the 'ubuntu-5' label while allowing packets for the other protocols (e.g., TCP and UDP). For this, we use 'matchProtocols' to define the protocol (i.e., ICMP) that we want to block.
Verification: After applying this policy, please get into the container with the 'ubuntu-5' label and run 'curl https://kubernetes.io/'. This will work fine. Then, run 'ping 8.8.8.8'. You will see 'Permission denied' since the 'ping' command internally uses the ICMP protocol.
Capabilities Restriction
Block Raw Sockets (i.e., non-TCP/UDP packets) ()
Explanation: We want to block any network operations using raw sockets from the containers with the 'ubuntu-1' label, meaning that containers cannot send non-TCP/UDP packets (e.g., ICMP echo request or reply) to other containers. To achieve this, we use matchCapabilities and specify the 'CAP_NET_RAW' capability to block raw socket creations inside the containers. Here, since we use the stream and datagram sockets to TCP and UDP packets respectively, we can still send those packets to others.
Verification: After applying this policy, please get into the container with the 'ubuntu-1' label and run 'curl https://kubernetes.io/'. This will work fine. Then, run 'ping 8.8.8.8'. You will see 'Operation not permitted' since the 'ping' command internally requires a raw socket to send ICMP packets.
System calls alerting
Alert for all unlink syscalls
Alert on all rmdir syscalls targeting anything in /home/ directory and sub-directories
Here is the specification of a Cluster security policy.
Note Please note that for system calls monitoring we only support audit action no matter what the value of action is
Now, we will briefly explain how to define a cluster security policy.
A cluster security policy starts with the base information such as apiVersion, kind, and metadata. The apiVersion would be the same in any security policies. In the case of metadata, you need to specify the names of a policy and a namespace where you want to apply the policy and kind would be KubeArmorClusterPolicy.
The severity part is somewhat important. You can specify the severity of a given policy from 1 to 10. This severity will appear in alerts when policy violations happen.
The tags part is optional. You can define multiple tags (e.g., WARNING, SENSITIVE, MITRE, STIG, etc.) to categorize security policies.
The message part is optional. You can add an alert message, and then the message will be presented in alert logs.
In the selector section for cluster-based policies, we use matchExpressions to define the namespaces where the policy should be applied. Currently, only namespaces can be matched, so the key should be 'namespace'. The operator will determine whether the policy should apply to the namespaces specified in the values field or not.
Operator: In When the operator is set to In, the policy will be applied only to the namespaces listed in the values field.
Operator: NotIn When the operator is set to NotIn, the policy will be applied to all other namespaces except those listed in the values field.
TIP If the selector operator is omitted in the policy, it will be applied across all namespaces.
In each match, there are three options.
ownerOnly (static action: allow owner only; otherwise block all)
If this is enabled, the owners of the executable(s) defined with matchPaths and matchDirectories will be only allowed to execute.
recursive
If this is enabled, the coverage will extend to the subdirectories of the directory defined with matchDirectories.
fromSource
If a path is specified in fromSource, the executable at the path will be allowed/blocked to execute the executables defined with matchPaths or matchDirectories. For better understanding, let us say that an operator defines a policy as follows. Then, /bin/bash will be only allowed (blocked) to execute /bin/sleep. Otherwise, the execution of /bin/sleep will be blocked (allowed).
The file section is quite similar to the process section.
The only difference between 'process' and 'file' is the readOnly option.
readOnly (static action: allow to read only; otherwise block all)
If this is enabled, the read operation will be only allowed, and any other operations (e.g., write) will be blocked.
In the case of network, there is currently one match type: matchProtocols. You can define specific protocols among TCP, UDP, and ICMP.
In the case of syscalls, there are two types of matches, matchSyscalls and matchPaths. matchPaths can be used to target system calls targeting specific binary path or anything under a specific directory, additionally you can slice based on syscalls generated by a binary or a group of binaries in a directory. You can use matchSyscall as a more general rule to match syscalls from all sources or from specific binaries.
There is one options in each match.
fromSource
If a path is specified in fromSource, kubearmor will match only syscalls generated by the defined source. For better undrestanding, lets take the example below. Only unlink system calls generated by /bin/bash will be matched.
recursive
If this is enabled, the coverage will extend to the subdirectories of the directory.
Action
Here, we demonstrate how to define a cluster security policies.
Process Execution Restriction
Block a specific executable - In operator ()
Explanation: The purpose of this policy is to block the execution of '/usr/bin/apt' in the containers present in the namespace nginx1. For this, we define the 'nginx1' value and operator as 'In' in selector -> matchExpressions and the specific path ('/usr/bin/apt') in process -> matchPaths. Also, we put 'Block' as the action of this policy.
Verification: After applying this policy, please get into one of the containers in the namespace 'nginx1' (using "kubectl -n nginx1 exec -it nginx-X-... -- bash") and run '/usr/bin/apt'. You will see that /usr/bin/apt is blocked.
Block a specific executable - NotIn operator()
Explanation: The purpose of this policy is to block the execution of '/usr/bin/apt' in all containers present in the cluster except that are in the namespace nginx1. For this, we define the 'nginx1' value and operator as 'NotIn' in selector -> matchExpressions and the specific path ('/usr/bin/apt') in process -> matchPaths. Also, we put 'Block' as the action of this policy.
Verification: After applying this policy, please get into one of the containers in the namespace 'nginx1' (using "kubectl -n nginx1 exec -it nginx-X-... -- bash") and run '/usr/bin/apt'. You will see that /usr/bin/apt is not blocked. Now try running same command in container inside 'nginx2' namespace and it should not be blocked.
File Access Restriction
Block accessing specific file ()
Explanation: The purpose of this policy is to block the container within the namespace 'nginx2' to read '/etc/host.conf'.
Verification: After applying this policy, please get into the container within the namespace 'nginx2' and run 'cat /etc/host.conf'. You can see the operation is blocked.
Block accessing specific file ()
Explanation: The purpose of this policy is to block read access for '/etc/host.conf' in all the containers except the namespace 'bginx2'.
Verification: After applying this policy, please get into the container within the namespace 'nginx2' and run 'cat /etc/host.conf'. You can see the operation is not blocked and can see the content of the file. Now try to run 'cat /etc/host.conf' in container of 'nginx1' namespace, this operation should be blocked.
Note Other operations like Network, Capabilities, Syscalls also behave in same way as in security policy. The difference only lies in how we match the cluster policy with the namespaces.
KubeArmor maintainers welcome individuals and organizations from across the cloud security landscape (creators and implementers alike) to make contributions to the project. We equally value the addition of technical contributions and enhancements of documentation that helps us grow the community and strengthen the value of KubeArmor. We invite members of the community to contribute to the project!
To make a contribution, please follow the steps below.
Fork this repository (KubeArmor)
First, fork this repository by clicking on the Fork button (top right).
Then, click your ID on the pop-up screen.
This will create a copy of KubeArmor in your account.
Clone the repository
Now clone Kubearmor locally into your dev environment.
This will clone a copy of Kubearmor installed in your dev environment.
Make changes
First, go into the repository directory and make some changes.
Please refer to to set up your environment for KubeArmor contribution.
Check the changes
Please run "test_kubearmor.sh" before committing the changes
If you see any warnings or errors, please fix them first.
Commit changes
Please see your changes using "git status" and add them to the branch using "git add".
Then, commit the changes using the "git commit" command.
Please make sure that your changes are properly tested on your machine.
Push changes to your forked repository
Push your changes using the "git push" command.
Create a pull request with your changes with the following steps
First, go to your repository on GitHub.
Then, click "Pull request" button.
After checking your changes, click 'Create pull request'.
A pull request should contain the details of all commits as specific as possible, including "Fixes: #(issue number)".
Finally, click the "Create pull request" button.
The changes would be merged post a review by the respective module owners. Once the changes are merged, you will get a notification, and the corresponding issue will be closed.
DCO Signoffs
To ensure that contributors are only submitting work that they have rights to, we are requiring everyone to acknowledge this by signing their work. Any copyright notices in this repo should specify the authors as "KubeArmor authors".
To sign your work, just add a line like this at the end of your commit message:
This can easily be done with the -s or --signoff option to git commit.
By doing this, you state that the source code being submitted originated from you (see https://developercertificate.org).
There are two ways to check the functionalities of KubeArmor: 1) testing KubeArmor manually and 2) using the testing framework.
To install the controller from KubeArmor docker repository to your cluster run
To install the controller (local version) to your cluster run
Beforehand, check if the KubeArmorPolicy and KubeArmorHostPolicy CRDs are already applied.
If they are still not applied, do so.
Now you can apply specific policies.
flags:
Note that you will see alerts and logs generated right after karmor runs logs; thus, we recommend to run the above command in other terminal to see logs live.
The auto-testing framework operates based on two things: microservices and test scenarios for each microservice.
Microservices
Then, create YAML files for the microservice
Test scenarios
Then, define a YAML file for a test policy in the directory
Create cmd files whose names are like 'cmd#'
Here is a template for a cmd file.
This is a cmd example of a test scenario.
The case that KubeArmor is directly running in a host
Compile KubeArmor
Run the auto-testing framework
Check the test report
The case that KubeArmor is running as a daemonset in Kubernetes
Run the testing framework
Check the test report
Security Policy for Pods/Containers [] []
Cluster level security Policy for Pods/Containers [] []
Security Policy for Hosts/Nodes [] [] ...
,
Minutes:
Calendar invite: ,
KubeArmor uses 's system call utility functions.
KubeArmor is of the Cloud Native Computing Foundation.
KubeArmor roadmap is tracked via
In the process section, there are three types of matches: matchPaths, matchDirectories, and matchPatterns. You can define specific executables using matchPaths or all executables in specific directories using matchDirectories. In the case of matchPatterns, advanced operators may be able to determine particular patterns for executables by using regular expressions. However, the coverage of regular expressions is highly dependent on AppArmor (). Thus, we generally do not recommend using this match.
In the case of capabilities, there is currently one match type: matchCapabilities. You can define specific capability names to allow or block using matchCapabilities. You can check available capabilities in .
The action could be Allow, Audit, or Block. Security policies would be handled in a blacklist manner or a whitelist manner according to the action. Thus, you need to define the action carefully. You can refer to for more details. In the case of the Audit action, we can use this action for policy verification before applying a security policy with the Block action. For System calls monitoring, we only support audit mode no matter what the action is set to.
UEK R7 can be installed on OL 8.6 by following the easy-to-follow instructions provided here in this .
The KubeArmor team has brought this to the attention of the on StackOverflow and await their response.
For more such differences checkout .
After this, exit out of the node shell and follow the .
So we need to enable for Kubearmor to apply and enforce policies as expected.
You can also enable AppArmor if you want to use it as a security module to enforce KubeArmor policies, please refer . There is a chance that neither AppArmor nor BPF-LSM is enabled on some nodes.
You can refer to security policies defined for example microservices in .
Watch alerts using cli tool
Create a directory for a microservice in
As an example, we created 'multiubuntu' in and defined 'multiubuntu-deployment.yaml' in .
Create a directory whose name is like '[microservice name]_[scenario name]' in
You can refer to predefined testcases in .
KubeArmor helps organizations enforce a zero trust posture within their Kubernetes clusters. It allows users to define an allow-based policy that allows specific operations, and denies or audits all other operations. This helps to ensure that only authorized activities are allowed within the cluster, and that any deviations from the expected behavior are denied and flagged for further investigation.
By implementing a zero trust posture with KubeArmor, organizations can increase their security posture and reduce the risk of unauthorized access or activity within their Kubernetes clusters. This can help to protect sensitive data, prevent system breaches, and maintain the integrity of the cluster.
KubeArmor supports allow-based policies which results in specific actions to be allowed and denying/auditing everything else. For example, a specific pod/container might only invoke a set of binaries at runtime. As part of allow-based rules you can specify the set of processes that are allowed and everything else is either audited or denied based on the default security posture.
Install the nginx deployment using
kubectl create deployment nginx --image=nginx.
Set the default security posture to default-deny.
kubectl annotate ns default kubearmor-file-posture=block --overwrite
Apply the following policy:
Observe that the policy contains Allow action. Once there is any KubeArmor policy having Allow action then the pods enter least permissive mode, allowing only explicitly allowed operations.
Note: Use kubectl port-forward $POD --address 0.0.0.0 8080:80 to access nginx and you can see that the nginx web access still works normally.
Lets try to execute some other processes:
This would be permission denied.
Achieving Zero Trust Security Posture is difficult. However, the more difficult part is to maintain the Zero Trust posture across application updates. There is also a risk of application downtime if the security posture is not correctly identified. While KubeArmor provides a way to enforce Zero Trust Security Posture, identifying the policies/rules for achieving this is non-trivial and requires that you keep the policies in dry-run mode (or default audit mode) before using the default-deny mode.
KubeArmor provides framework so as to smoothen the journey to Zero Trust posture. For e.g., it is possible to set dry-run/audit mode at the namespace level by configuring security posture. Thus, you can have different namespaces in different default security posture modes (default-deny vs default-audit). Users can switch to default-deny mode once they are comfortable (i.e., they do not see any alerts) with the settings.
KubeArmor is a security solution for the Kubernetes and cloud native platforms that helps protect your workloads from attacks and threats. It does this by providing a set of hardening policies that are based on industry-leading compliance and attack frameworks such as CIS, MITRE, NIST-800-53, and STIGs. These policies are designed to help you secure your workloads in a way that is compliant with these frameworks and recommended best practices.
One of the key features of KubeArmor is that it provides these hardening policies out-of-the-box, meaning that you don't have to spend time researching and configuring them yourself. Instead, you can simply apply the policies to your workloads and immediately start benefiting from the added security that they provide.
Additionally, KubeArmor presents these hardening policies in the context of your workload, so you can see how they will be applied and what impact they will have on your system. This allows you to make informed decisions about which policies to apply, and helps you understand the trade-offs between security and functionality.
Overall, KubeArmor is a powerful tool for securing your Kubernetes workloads, and its out-of-the-box hardening policies based on industry-leading compliance and attack frameworks make it easy to get started and ensure that your system is as secure as possible.
Hardening policies are derived from industry leading compliance standards and attack frameworks such as CIS, MITRE, NIST, STIGs, and several others. KubeArmor Policy Templates contains the latest hardening policies. KubeArmor client tool (karmor) provides a way (karmor recommend) to fetch the policies in the context of the kubernetes workloads or specific container using command line. The output is a set of KubeArmorPolicy or KubeArmorHostPolicy that can be applied using k8s native tools (such as kubectl apply).
The rules in hardening policies are based on inputs from:
Several others...
Pre-requisites:
Install KubeArmor
curl -sfL http://get.kubearmor.io/ | sudo sh -s -- -b /usr/local/bin && karmor install
Get the hardening policies in context of all the deployment in namespace NAMESPACE:
karmor recommend -n NAMESPACE
The recommended policies would be available in the out folder.
Key highlights:
The hardening policies are available by default in the out folder separated out in directories based on deployment names.
Get an HTML report by using the option --report report.html with karmor recommend.
Get hardening policies in context to specific compliance by specifying --tag <CIS/MITRE/...> option.
This recipe explains how to use KubeArmor directly on a VM/Bare-Metal machine, and we tested the following steps on Ubuntu hosts.
The recipe installs kubearmor as systemd process and karmor cli tool to manage policies and show alerts/telemetry.
Download the latest release or KubeArmor.
Install KubeArmor (VER is the kubearmor release version)
Note that the above command doesn't installs the recommended packages, as we ship object files along with the package file. In case you don't have BTF, consider removing
--no-install-recommendsflag.
Check the status of KubeArmor using sudo systemctl status kubearmor or use sudo journalctl -u kubearmor -f to continuously monitor kubearmor logs.
Following policy is to deny execution of sleep binary on the host:
Save the above policy to hostpolicy.yaml and apply:
Now if you run sleep command, the process would be denied execution.
Note that
sleepmay not be blocked if you run it in the same terminal where you apply the above policy. In that case, please open a new terminal and runsleepagain to see if the command is blocked.
KubeArmor supports attack prevention, not just observability and monitoring. More importantly, the prevention is handled inline: even before a process is spawned, a rule can deny execution of a process. Most other systems typically employ "post-attack mitigation" that kills a process/pod after malicious intent is observed, allowing an attacker to execute code on the target environment. Essentially KubeArmor uses inline mitigation to reduce the attack surface of a pod/container/VM. KubeArmor leverages best of breed Linux Security Modules (LSMs) such as AppArmor, BPF-LSM, and SELinux (only for host protection) for inline mitigation. LSMs have several advantages over other techniques:
KubeArmor does not change anything with the pod/container.
KubeArmor does not require any changes at the host level or at the CRI (Container Runtime Interface) level to enforce blocking rules. KubeArmor deploys as a non-privileged DaemonSet with certain capabilities that allows it to monitor other pods/containers and the host.
A given cluster can have multiple nodes utilizing different LSMs. KubeArmor abstracts away complexities of LSMs and provides an easy way to enforce policies. KubeArmor manages complexity of LSMs under-the-hood.
Post-exploit Mitigation works by killing a suspicious process in response to an alert indicating malicious intent.
Attacker is allowed to execute a binary. Attacker could disable security controls, access logs, etc to circumvent attack detection.
By the time a malicious process is killed, sensitive contents could have already been deleted, encrypted, or transmitted.
Quoting Grsecurity, “post-exploitation detection/mitigation is at the mercy of an exploit writer putting little to no effort into avoiding tripping these detection mechanisms.”
Pod Security Context allows one to specify native AppArmor or native SELinux policies.
This approach has multiple problems:
It is often difficult to predict which LSM (AppArmor or SELinux) would be available on the target node.
BPF-LSM is not supported by Pod Security Context.
It is difficult to manually specify an AppArmor or SELinux policy. Changing default AppArmor or SELinux policies might result in more security holes since it is difficult to decipher the implications of the changes and can be counter-productive.
Different managed cloud providers use different default distributions. Google GKE COS uses AppArmor by default, AWS Bottlerocket uses BPF-LSM and SELinux, and AWS Amazon Linux 2 uses only SELinux by default. Thus it is challenging to use Pod Security Context in multi-cloud deployments.
References:
KubeArmor supports following types of workloads:
K8s orchestrated: Workloads deployed as k8s orchestrated containers. In this case, Kubearmor is deployed as a k8s daemonset. Note, KubeArmor supports policy enforcement on both k8s-pods (KubeArmorPolicy) as well as k8s-nodes (KubeArmorHostPolicy).
Containerized: Workloads that are containerized but not k8s orchestrated are supported. KubeArmor installed in systemd mode can be used to protect such workloads.
VM/Bare-Metals: Workloads deployed on Virtual Machines or Bare Metal i.e. workloads directly operating as host/system processes. In this case, Kubearmor is deployed in systemd mode.
Provider
K8s engine
OS Image
Arch
Audit Rules
Blocking Rules
LSM Enforcer
Remarks
Onprem
x86_64, ARM
x86_64
Ubuntu >= 16.04
x86_64
Microsoft
Ubuntu >= 18.04
x86_64
Oracle
x86_64
IBM
Ubuntu
x86_64
AWS
Amazon Linux 2 (kernel >=5.8)
x86_64
AWS
Amazon Linux 2 (kernel <=5.4)
x86_64
SELinux
AWS
Ubuntu
x86_64
AppArmor
AWS
x86_64
AWS
Ubuntu
ARM
AppArmor
AWS
Amazon Linux 2
ARM
SELinux
RedHat
x86_64
SELinux
RedHat
x86_64
RedHat
x86_64
Rancher
x86_64
Rancher
x86_64
Oracle
ARM
SELinux
VMware
TBD
x86_64
Mirantis
Ubuntu>=20.04
x86_64
AppArmor
Digital Ocean
Debian GNU/Linux 11 (bullseye)
x86_64
Alibaba Cloud
Alibaba Cloud Linux 3.2104 LTS
x86_64
Following distributions are tested for VM/Bare-metal based installations:
SUSE
SUSE Enterprise 15
Full
Full
Debian
Full
Full
Ubuntu
18.04 / 16.04 / 20.04
Full
Full
RedHat / CentOS
RHEL / CentOS <= 8.4
Full
Partial
RedHat / CentOS
RHEL / CentOS >= 8.5
Full
Full
Fedora
Fedora 34 / 35
Full
Full
Rocky Linux
Rocky Linux >= 8.5
Full
Full
AWS
Amazon Linux 2022
Full
Full
AWS
Amazon Linux 2023
Full
Full
RaspberryPi (ARM)
Debian
Full
Full
ArchLinux
ArchLinux-6.2.1
Full
Full
Alibaba
Alibaba Cloud Linux 3.2104 LTS 64 bit
Full
Full
Note Full: Supports both enforcement and observability Partial: Supports only observability
Please approach the Kubearmor community on slack or raise a GitHub issue to express interest in adding the support.
It would be very much appreciated if you can test kubearmor on a platform not listed above and if you have access to. Once tested you can update this document and raise a PR.
KubeArmor currently supports enabling visibility for containers and hosts.
Visibility for hosts is not enabled by default, however it is enabled by default for containers .
The karmor tool provides access to both using karmor logs.
If you don't have access to a K8s cluster, please follow to set one up.
karmor CLI tool:
To deploy app follow
Now we need to deploy some sample policies
This sample policy blocks execution of the apt and apt-get commands in wordpress pods with label selector app: wordpress.
Checking default visibility
Container visibility is enabled by default. We can check it using kubectl describe and grep kubearmor-visibility
For pre-existing workloads : Enable visibility using kubectl annotate. Currently KubeArmor supports process, file, network, capabilities
Open up a terminal, and watch logs using the karmor cli
In another terminal, simulate a policy violation . Try sleep inside a pod
In the terminal running karmor logs, the policy violation along with container visibility is shown, in this case for example
The logs can also be generated in JSON format using karmor logs --json
Host Visibility is not enabled by default . To enable Host Visibility we need to annotate the node using kubectl annotate node
To confirm it use kubectl describe and grep kubearmor-visibility
Now we can get general telemetry events in the context of the host using karmor logs .The logs related to Host Visibility will have type Type: HostLogand Operation: File | Process | Network
KubeArmor has the ability to let the user select what kind of events have to be traced by changing the annotation kubearmor-visibility at the namespace.
Checking Namespace visibility
Namespace visibility can be checked using kubectl describe.
To update the visibility of namespace : Now let's update Kubearmor visibility using kubectl annotate. Currently KubeArmor supports process, file, network, capabilities. Lets try to update visibility for the namespace wordpress-mysql
Note: To turn off the visibility across all aspects, use
kubearmor-visibility=none. Note that any policy violations or events that results in non-success returns would still be reported in the logs.
Open up a terminal, and watch logs using the karmor cli
In another terminal, let's exec into the pod and run some process commands . Try ls inside the pod
Now, we can notice that no logs have been generated for the above command and logs with only Operation: Network are shown.
Note If telemetry is disabled, the user wont get audit event even if there is an audit rule.
Note Only the logs are affected by changing the visibility, we still get all the alerts that are generated.
Let's simulate a sample policy violation, and see whether we still get alerts or not.
Policy violation :
Here, note that the alert with Operation: Process is reported.
KubeArmor supports configurable default security posture. The security posture could be allow/audit/deny. Default Posture is used when there's atleast one Allow policy for the given deployment i.e. KubeArmor is handling policies in whitelisting manner (more about this in ).
There are two default mode of operations available block and audit. block mode blocks all the operations that are not allowed in the policy. audit generates telemetry events for operations that would have been blocked otherwise.
KubeArmor has 4 types of resources: Process, File, Network and Capabilities. Default Posture is configurable for each of the resources seperately except Process. Process based operations are treated under File resource only.
Note By default, KubeArmor set the Global default posture to
audit
Global default posture is configured using configuration options passed to KubeArmor using configuration file
Or using command line flags with the KubeArmor binary
We use namespace annotations to configure default posture per namespace. Supported annotations keys are kubearmor-file-posture,kubearmor-network-posture and kubearmor-capabilities-posture with values block or audit. If a namespace is annotated with a supported key and an invalid value ( like kubearmor-file-posture=invalid), KubeArmor will update the value with the global default posture ( i.e. to kubearmor-file-posture=block).
Let's start KubeArmor with configuring default network posture to audit in the following YAML.
Contents of kubearmor.yaml
Here's a sample policy to allow tcp connections from curl binary.
Inside the ubuntu-5-deployment, if we try to access tcp using curl. It works as expected with no telemetry generated.
If we try to access udp using curl, a bunch of telemetry is generated for the udp access.
curl google.comrequires UDP for DNS resolution.
Generated alert has Policy Name DefaultPosture and Action as Audit
Now let's update the default network posture to block for multiubuntu namespace.
Now if we try to access udp using curl, the action is blocked and related alerts are generated.
Here curl couldn't resolve google.com due to blocked access to UDP.
Generated alert has Policy Name DefaultPosture and Action as Block
Let's try to set the annotation value to something invalid.
We can see that, annotation value was automatically updated to audit since that was global mode of operation for network in the KubeArmor configuration.
Here is the specification of a host security policy.
Note Please note that for system calls monitoring we only support audit action no matter what the value of action is
Now, we will briefly explain how to define a host security policy.
Common
A security policy starts with the base information such as apiVersion, kind, and metadata. The apiVersion and kind would be the same in any security policies. In the case of metadata, you need to specify the name of a policy.
Make sure that you need to use KubeArmorHostPolicy, not KubeArmorPolicy.
Severity
You can specify the severity of a given policy from 1 to 10. This severity will appear in alerts when policy violations happen.
Tags
The tags part is optional. You can define multiple tags (e.g., WARNING, SENSITIVE, MITRE, STIG, etc.) to categorize security policies.
Message
The message part is optional. You can add an alert message, and then the message will be presented in alert logs.
NodeSelector
The node selector part is relatively straightforward. Similar to other Kubernetes configurations, you can specify (a group of) nodes based on labels.
If you do not have any custom labels, you can use system labels as well.
Process
In the process section, there are three types of matches: matchPaths, matchDirectories, and matchPatterns. You can define specific executables using matchPaths or all executables in specific directories using matchDirectories. In the case of matchPatterns, advanced operators may be able to determine particular patterns for executables by using regular expressions. However, we generally do not recommend using this match.
In each match, there are three options.
ownerOnly (static action: allow owner only; otherwise block all)
If this is enabled, the owners of the executable(s) defined with matchPaths and matchDirectories will be only allowed to execute.
recursive
If this is enabled, the coverage will extend to the subdirectories of the directory defined with matchDirectories.
fromSource
If a path is specified in fromSource, the executable at the path will be allowed/blocked to execute the executables defined with matchPaths or matchDirectories. For better understanding, let us say that an operator defines a policy as follows. Then, /bin/bash will be only allowed (blocked) to execute /bin/sleep. Otherwise, the execution of /bin/sleep will be blocked (allowed).
File
The file section is quite similar to the process section.
The only difference between 'process' and 'file' is the readOnly option.
readOnly (static action: allow to read only; otherwise block all)
If this is enabled, the read operation will be only allowed, and any other operations (e.g., write) will be blocked.
Network
In the case of network, there is currently one match type: matchProtocols. You can define specific protocols among TCP, UDP, and ICMP.
Capabilities
Syscalls
In the case of syscalls, there are two types of matches, matchSyscalls and matchPaths. matchPaths can be used to target system calls targeting specific binary path or anything under a specific directory, additionally you can slice based on syscalls generated by a binary or a group of binaries in a directory. You can use matchSyscall as a more general rule to match syscalls from all sources or from specific binaries.
There is one options in each match.
fromSource If a path is specified in fromSource, kubearmor will match only syscalls generated by the defined source. For better undrestanding, lets take the example below. Only unlink system calls generated by /bin/bash will be matched.
recursive
If this is enabled, the coverage will extend to the subdirectories of the directory.
Action
The action could be Audit or Block in general. In order to use the Allow action, you should define 'fromSource'; otherwise, all Allow actions will be ignored by default.
If 'fromSource' is defined, we can use all actions for specific rules.
For System calls monitoring, we only support audit mode no matter what the action is set to.
KubeArmor has visibility into systems and application behavior. KubeArmor summarizes/aggregates the information and provides a user-friendly view to figure out the application behavior.
Process data:
What are the processes executing in the pods?
What processes are executing through which parent processes?
File data:
What are the file system accesses made by different processes?
Network Accesses:
What are the Ingress/Egress connections from the pod?
What server binds are done in the pod?
Get visibility into process executions in default namespace.
Native Json format (this document)
KubeArmor CEF Format (coming soon...)
Container alerts are generated when there is a policy violation or audit event that is raised due to a policy action. For example, a policy might block execution of a process. When the execution is blocked by KubeArmor enforcer, KubeArmor generates an alert event implying policy action. In the case of an Audit action, the KubeArmor will only generate an alert without actually blocking the action.
The primary difference in the container alerts events vs the telemetry events (showcased above) is that the alert events contains certain additional fields such as policy name because of which the alert was generated and other metadata such as "Tags", "Message", "Severity" associated with the policy rule.
The fields are self-explanatory and have similar meaning as in the context of container based events (explained above).
Here is the specification of a security policy.
Note Please note that for system calls monitoring we only support audit action no matter what the value of action is
Now, we will briefly explain how to define a security policy.
A security policy starts with the base information such as apiVersion, kind, and metadata. The apiVersion and kind would be the same in any security policies. In the case of metadata, you need to specify the names of a policy and a namespace where you want to apply the policy.
The severity part is somewhat important. You can specify the severity of a given policy from 1 to 10. This severity will appear in alerts when policy violations happen.
The tags part is optional. You can define multiple tags (e.g., WARNING, SENSITIVE, MITRE, STIG, etc.) to categorize security policies.
The message part is optional. You can add an alert message, and then the message will be presented in alert logs.
The selector part is relatively straightforward. Similar to other Kubernetes configurations, you can specify (a group of) pods based on labels.
In each match, there are three options.
ownerOnly (static action: allow owner only; otherwise block all)
If this is enabled, the owners of the executable(s) defined with matchPaths and matchDirectories will be only allowed to execute.
recursive
If this is enabled, the coverage will extend to the subdirectories of the directory defined with matchDirectories.
fromSource
If a path is specified in fromSource, the executable at the path will be allowed/blocked to execute the executables defined with matchPaths or matchDirectories. For better understanding, let us say that an operator defines a policy as follows. Then, /bin/bash will be only allowed (blocked) to execute /bin/sleep. Otherwise, the execution of /bin/sleep will be blocked (allowed).
The file section is quite similar to the process section.
The only difference between 'process' and 'file' is the readOnly option.
readOnly (static action: allow to read only; otherwise block all)
If this is enabled, the read operation will be only allowed, and any other operations (e.g., write) will be blocked.
In the case of network, there is currently one match type: matchProtocols. You can define specific protocols among TCP, UDP, and ICMP.
In the case of syscalls, there are two types of matches, matchSyscalls and matchPaths. matchPaths can be used to target system calls targeting specific binary path or anything under a specific directory, additionally you can slice based on syscalls generated by a binary or a group of binaries in a directory. You can use matchSyscall as a more general rule to match syscalls from all sources or from specific binaries.
There is one options in each match.
fromSource
If a path is specified in fromSource, kubearmor will match only syscalls generated by the defined source. For better undrestanding, lets take the example below. Only unlink system calls generated by /bin/bash will be matched.
recursive
If this is enabled, the coverage will extend to the subdirectories of the directory.
Action
Here, we demonstrate how to define host security policies.
Process Execution Restriction
Block a specific executable ()
Explanation: The purpose of this policy is to block the execution of '/usr/bin/diff' in a host whose host name is 'kubearmor-dev'. For this, we define 'kubernetes.io/hostname: kubearmor-dev' in nodeSelector -> matchLabels and the specific path ('/usr/bin/diff') in process -> matchPaths. Also, we put 'Block' as the action of this policy.
Verification: After applying this policy, please open a new terminal (or connect to the host with a new session) and run '/usr/bin/diff'. You will see that /usr/bin/diff is blocked.
NOTE
The given policy works with almost every linux distribution. If it is not working in your case, check the process location. The following location shows location of sleep binary in different ubuntu distributions:
In case of Ubuntu 20.04 : /usr/bin/sleep
In case of Ubuntu 18.04 : /bin/sleep
File Access Restriction
Audit a critical file access ()
Explanation: The purpose of this policy is to audit any accesses to a critical file (i.e., '/etc/passwd'). Since we want to audit one critical file, we use matchPaths to specify the path of '/etc/passwd'.
Verification: After applying this policy, please open a new terminal (or connect to the host with a new session) and run 'sudo cat /etc/passwd'. Then, check the alert logs of KubeArmor.
System calls alerting
Alert for all unlink syscalls
Alert on all rmdir syscalls targeting anything in /home/ directory and sub-directories
Note Skip the steps for the vagrant setup if you're directly compiling KubeArmor on the Linux host. Proceed to setup K8s on the same host by resolving any dependencies.
Requirements
Here is the list of requirements for a Vagrant environment
Clone the KubeArmor github repository in your system
Install Vagrant and VirtualBox in your environment, go to the vagrant path and run the setup.sh file
VM Setup using Vagrant
Now, it is time to prepare a VM for development.
To create a vagrant VM
Output will show up as ...
To get into the vagrant VM
Output will show up as ...
To destroy the vagrant VM
VM Setup using Vagrant with Ubuntu 21.10 (v5.13)
To use the recent Linux kernel v5.13 for dev env, you can run make with the NETNEXT flag set to 1 for the respective make option.
You can also make the setting static by changing NETNEXT=0 to NETNEXT=1 in the Makefile.
Requirements
Here is the list of minimum requirements for self-managed Kubernetes.
Alternative Setup
You can try the following alternative if you face any difficulty in the above Kubernetes (kubeadm) setup.
Note Please make sure to set up the alternative k8s environment on the same host where the KubeArmor development environment is running.
K3s
MicroK8s
No Support - Docker Desktops
KubeArmor does not work with Docker Desktops on Windows and macOS because KubeArmor integrates with Linux-kernel native primitives (including LSMs).
Development Setup
In order to install all dependencies, please run the following command.
Now, you are ready to develop any code for KubeArmor. Enjoy your journey with KubeArmor.
Compilation
Check if KubeArmor can be compiled on your environment without any problems.
If you see any error messages, please let us know the issue with the full error messages through #kubearmor-development channel on CNCF slack.
Execution
In order to directly run KubeArmor in a host (not as a container), you need to run a local proxy in advance.
Then, run KubeArmor on your environment.
Note If you have followed all the above steps and still getting the warning
The node information is not available, then this could be due to the case-sensitivity discrepancy in the actual hostname (obtained by runninghostname) and the hostname used by Kubernetes (underkubectl get nodes -o wide). K8s converts the hostname to lowercase, which results in a mismatch with the actual hostname. To resolve this, change the hostname to lowercase using the commandhostnamectl set-hostname <lowercase-hostname>.
KubeArmor Controller
Starting from KubeArmor v0.11 - annotations, container policies, and host policies are handled via kubearmor controller, the controller code can be found under pkg/KubeArmorController.
To install the controller from KubeArmor docker repository run
To install the controller (local version) to your cluster run
if you need to setup a local registry to push you image, use docker-registry.sh script under ~/KubeArmor/contribution/local-registry directory
Here, we briefly give you an overview of KubeArmor's directories.
Source code for KubeArmor (/KubeArmor)
Source code for KubeArmor Controller (CRD)
Deployment tools and files for KubeArmor
Files for testing
Protect critical paths such as cert bundles MITRE, STIGs, CIS based rules Restrict access to raw DB table
Process Whitelisting Network Whitelisting Control access to sensitive assets
Process execs, File System accesses Service binds, Ingress, Egress connections Sensitive system call profiling
Kubernetes Deployment Containerized Deployment VM/Bare-Metal Deployment
kubeadm, , , microk8s
, AppArmor
, AppArmor
All
, AppArmor
All
, AppArmor
>=7
, AppArmor
<=8.4
>=8.5
>=9.2
, AppArmor
, AppArmor
/
Note: This example is in the environment.
For better understanding, you can check .
In the case of capabilities, there is currently one match type: matchCapabilities. You can define specific capability names to allow or block using matchCapabilities. You can check available capabilities in .
For better understanding, you can check .
In the process section, there are three types of matches: matchPaths, matchDirectories, and matchPatterns. You can define specific executables using matchPaths or all executables in specific directories using matchDirectories. In the case of matchPatterns, advanced operators may be able to determine particular patterns for executables by using regular expressions. However, the coverage of regular expressions is highly dependent on AppArmor (). Thus, we generally do not recommend using this match.
In the case of capabilities, there is currently one match type: matchCapabilities. You can define specific capability names to allow or block using matchCapabilities. You can check available capabilities in .
The action could be Allow, Audit, or Block. Security policies would be handled in a blacklist manner or a whitelist manner according to the action. Thus, you need to define the action carefully. You can refer to for more details. In the case of the Audit action, we can use this action for policy verification before applying a security policy with the Block action. For System calls monitoring, we only support audit mode no matter what the action is set to.
KubeArmor is designed for Kubernetes environment. If Kubernetes is not setup yet, please refer to . KubeArmor leverages CRI (Container Runtime Interfaces) APIs and works with Docker or Containerd or CRIO based container runtimes. KubeArmor uses LSMs for policy enforcement; thus, please make sure that your environment supports LSMs (either AppArmor or bpf-lsm). Otherwise, KubeArmor will operate in Audit-Mode with no policy "enforcement" support.
You can also develop and test KubeArmor on K3s instead of the self-managed Kubernetes. Please follow the instructions in .
You can also develop and test KubeArmor on MicroK8s instead of the self-managed Kubernetes. Please follow the instructions in .
will automatically install , , , and some other dependencies.
Action
specifies the action of the policy it has matched.
Audit/Block
ClusterName
gives information about the cluster for which the alert was generated
aks-test-cluster
Operation
gives details about what type of operation happened in the pod
File/Process/Network
ContainerID
information about the container ID where the policy violation or alert got generated
e10d5edb62ac2daa4eb9a2146e2f2cfa87b6a5f30bd3a
ContainerImage
shows the image that was used to spin up the container
docker.io/library/mysql:5.6@sha256:20575ecebe6216036d25dab5903808211f
ContainerName
specifies the Container name where the alert got generated
mysql
Data
shows the system call that was invoked for this operation
syscall=SYS_EXECVE
Enforcer
it specifies the name of the LSM that has enforced the policy
AppArmor/BPFLSM
HostName
shows the node name where the alert got generated
aks-agentpool-16128849-vmss000001
HostPID
gives the host Process ID
3647533
HostPPID
list the details of host Parent Process ID
3642706
Labels
shows the pod label from where alert generated
app=mysql
Message
gives the message specified in the policy
Alert! Execution of package management process inside container is denied
NamespaceName
lists the namespace where pod is running
wordpress-mysql
PID
lists the process ID running in container
266
PPID
lists the Parent process ID running in container
251
ParentProcessName
gives the parent process name from where the operation happend
/bin/bash
PodName
lists the pod name where the alert got generated
mysql-76ddc6ddc4-h47hv
PolicyName
gives the policy that was matched for this alert generation
harden-mysql-pkg-mngr-exec
ProcessName
specifies the operation that happened inside the pod for this alert
/usr/bin/apt
Resource
lists the resources that was requested
/usr/bin/apt
Result
shows whether the event was allowed or denied
Permission denied
Severity
gives the severity level of the operation
5
Source
lists the source from where the operation request came
/bin/bash
Tags
specifies the list of benchmarks this policy satisfies
NIST,NIST_800-53_CM-7(4),SI-4,process,NIST_800-53_SI-4
Timestamp
gives the details of the time this event tried to happen
1687868507
Type
shows whether policy matched or default posture alert
MatchedPolicy
UpdatedTime
gives the time of this alert
2023-06-27T12:21:47.932526
cluster_id
specifies the cluster id where the alert was generated
596
component_name
gives the component which generated this log/alert
kubearmor
tenant_id
specifies the tenant id where this cluster is onboarded in AccuKnox SaaS
11
ClusterName
gives information about the cluster for which the log was generated
default
Operation
gives details about what type of operation happened in the pod
File/Process/ Network
ContainerID
information about the container ID from where log was generated
7aca8d52d35ab7872df6a454ca32339386be
ContainerImage
shows the image that was used to spin up the container
docker.io/accuknox/knoxautopolicy:v0.9@sha256:bb83b5c6d41e0d0aa3b5d6621188c284ea
ContainerName
specifies the Container name where the log got generated
discovery-engine
Data
shows the system call that was invoked for this operation
syscall=SYS_OPENAT fd=-100 flags=O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC
HostName
shows the node name where the log got generated
aks-agentpool-16128849-vmss000001
HostPID
gives the host Process ID
967872
HostPPID
list the details of host Parent Process ID
967496
Labels
shows the pod label from where log generated
app=discovery-engine
Message
gives the message specified in the policy
Alert! Execution of package management process inside container is denied
NamespaceName
lists the namespace where pod is running
accuknox-agents
PID
lists the process ID running in container
1
PPID
lists the Parent process ID running in container
967496
ParentProcessName
gives the parent process name from where the operation happend
/usr/bin/containerd-shim-runc-v2
PodName
lists the pod name where the log got generated
mysql-76ddc6ddc4-h47hv
ProcessName
specifies the operation that happened inside the pod for this log
/knoxAutoPolicy
Resource
lists the resources that was requested
//accuknox-obs.db
Result
shows whether the event was allowed or denied
Passed
Source
lists the source from where the operation request came
/knoxAutoPolicy
Type
specifies it as container log
ContainerLog
This guide assumes you have access to a k8s cluster. If you want to try non-k8s mode, for instance systemd mode to protect/audit containers or processes on VMs/bare-metal, check here.
Check the KubeArmor support matrix to verify if your platform is supported.
You can find more details about helm related values and configurations here.
[!NOTE] kArmor CLI provides a Developer Friendly way to interact with KubeArmor Telemetry. You can stream KubeArmor telemetry independently of kArmor CLI tool and integrate it with your chosen SIEM (Security Information and Event Management) solutions. Here's a guide on how to achieve this integration. This guide assumes you have kArmor CLI to access KubeArmor Telemetry but you can view it on your SIEM tool once integrated.
[!NOTE]
$PODis used to refer to the target nginx pod in many cases below.
