From cf433b2eb3de94d3904a3256b3ce5b93df94a019 Mon Sep 17 00:00:00 2001 From: Asier Gutierrez Date: Mon, 12 Dec 2022 22:36:18 +0800 Subject: [PATCH 1/6] First KEP version for review --- keps/sig-node/asier/README.md | 530 ++++++++++++++++++++++++++++++++++ 1 file changed, 530 insertions(+) create mode 100644 keps/sig-node/asier/README.md diff --git a/keps/sig-node/asier/README.md b/keps/sig-node/asier/README.md new file mode 100644 index 00000000000..982fcc255d3 --- /dev/null +++ b/keps/sig-node/asier/README.md @@ -0,0 +1,530 @@ + + + + + +# KEP-XXXXX: IMA namespace support inside containers + + + + + + + + + + + + + +- [KEP-XXXXX: IMA namespace support inside containers](#kep-xxxxx-ima-namespace-support-inside-containers) + - [Release Signoff Checklist](#release-signoff-checklist) + - [Summary](#summary) + - [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) + - [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Story 3](#story-3) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) + - [Design Details](#design-details) + - [Linux kernel](#linux-kernel) + - [Runtime specification](#runtime-specification) + - [CRI API](#cri-api) + - [Kubernetes pod resource](#kubernetes-pod-resource) + - [Monitoring and alerting](#monitoring-and-alerting) + - [Test Plan](#test-plan) + - [Graduation Criteria](#graduation-criteria) + - [GA](#ga) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) + - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Implementation History](#implementation-history) + - [Drawbacks](#drawbacks) + - [Alternatives](#alternatives) + - [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + + + + +## Release Signoff Checklist + + + + + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + + + + + +[kubernetes.io]: https://kubernetes.io/ + +[kubernetes/enhancements]: https://git.k8s.io/enhancements + +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes + +[kubernetes/website]: https://git.k8s.io/website + + + +## Summary + + + +IMA namespaces allow to check the file integrity. This proposal adds file integrity inside containers deployed in kubernetes. + + + + +## Motivation + + + +File integrity is a way to improve security in systems allowing to: + +* Detect illicit activity + +* Detect unintended changes + +* Verify the status and health of the system + +* Comply with access rules + + + +This can be achieved using IMA (Integrity Measurement Architecture) and EVM (Extended Verification Module), which can use a TPM chip as Hardware Root of Trust for high security environments. + + + +### Goals + + + +* Allow IMA to work inside a container with remote attestation + + + +### Non-Goals + + + +## Proposal + +We propose to enable IMA linux namespaces in pods. + +Since IMA namespaces can be created when a container is launched, we can provide transparent integrity verification on any linux container. + +IMA and EVM can use a TPM chip as a hardware root of trust. Hence we can verify images against a set of golden hash values, as well as avoiding any further changes to the overlayfs to intercept calls and check the integrity of files. + + +### User Stories (Optional) + + + +#### Story 1 + +As a cluster admin, I want to detect undesired file changes, so that I can take out pods that have been compromised. + + + +#### Story 2 + + +As a cluster admin, I want to deploy a only proven and certified pods, so that I can comply with internal policies as well as security regulations. + +#### Story 3 + + +As a cluster admin, I want to deny access to certain files inside the pod, so that a potential intruder can't access sensitive information. + +### Notes/Constraints/Caveats (Optional) + + + +We need to enable IMA in the kernel and container runtimes (runC, CRI-O, docker, containerd, etc.). + + +### Risks and Mitigations + + + +## Design Details + + + + + +In order for the this feature to work, the nodes where the pods will be deploy should have IMA enabled and a recent kernel that supports IMA namespaces (WIP, it should be merged soon). The nodes should also have a TPM chip. We could use nodAffinity of labels and annotations in nodes in order to select where to deploy the pods. + +The linux kernel IMA namespace support is based on user namespaces. Therefore, the container runtime should first create a user namespace and then create an IMA namespaces. In order to use IMA namespaces it is necessary to enable user namespaces as well. + +Should we enable IMA namespaces by default when enabling user namespaces? + + +There will be a CRI API change which will allow the pod to use IMA namespaces and specify the namespace policy. + + + +### Linux kernel + + + +IMA is only available in Linux hosts and Linux containers. Unfortunately, IMA is not a separate namespace, which is needed in order to isolate it and be used inside containers. Upcoming kernel patches should add support for IMA namespaces. + + + +### Runtime specification + +In order to run a container we need a bundle that contains a config.json file with all the configuration. There is a Linux specific configuration section where the namespaces are listed. According to the standard, the following namespace types SHOULD be supported. + + + +* pid + +* network + +* mount + +* ipc + +* uts + +* user + +* cgroup + + + +We suggest to add a new namespaces, initially as a OPTIONAL type, in order to keep the backward compatibility. + + + +### CRI API + + + +We propose to add the following message. + + + +```protobuf + +message NamespaceOptions { + bool ima = 6; +} + +``` +### Kubernetes pod resource +We propose the following change in the podTemplate to enable IMA namespaces. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: nginx +spec: + securityContext: + ima: true + containers: + - name: nginx + image: nginx:1.14.2 + ports: + - containerPort: 80 +``` + +The item pod.spec.ima.policy will automatically enable IMA for all the container in the pod with a given policy. Since all containers in a pod share the same namespaces, we need to have this policy in advance when creating the pod and its infrastructure container. + + +### Monitoring and alerting + +This features will integrate with a future remote attestation procedure, which will monitor pods and in case of a violation take some actions like pod revocation, alerting, etc. + +### Test Plan + + + +Which unit tests should we include? + + + +### Graduation Criteria + + + +This KEP is a policy KEP, not a feature KEP. It will start as GA. + + + +#### GA + + + +### Upgrade / Downgrade Strategy + + + +### Version Skew Strategy + +## Production Readiness Review Questionnaire + + + +Not applicable because this is a policy KEP. + + + +## Implementation History + + + + + + + +## Drawbacks + + + + + + + +## Alternatives + + + + + + + +## Infrastructure Needed (Optional) + + + + From b1bffa27d8503c7f777725ffb314d3b71d88ba85 Mon Sep 17 00:00:00 2001 From: Asier Gutierrez Date: Thu, 22 Dec 2022 21:32:53 +0800 Subject: [PATCH 2/6] Initial draft for IMA namespace support in kubernetes --- keps/sig-node/{asier => 3702-ima-namespace-support}/README.md | 4 ++-- out | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) rename keps/sig-node/{asier => 3702-ima-namespace-support}/README.md (99%) create mode 100644 out diff --git a/keps/sig-node/asier/README.md b/keps/sig-node/3702-ima-namespace-support/README.md similarity index 99% rename from keps/sig-node/asier/README.md rename to keps/sig-node/3702-ima-namespace-support/README.md index 982fcc255d3..0496823c4f4 100644 --- a/keps/sig-node/asier/README.md +++ b/keps/sig-node/3702-ima-namespace-support/README.md @@ -121,7 +121,7 @@ SIG Architecture for cross-cutting KEPs). --> -# KEP-XXXXX: IMA namespace support inside containers +# KEP-3702: IMA namespace support inside containers @@ -130,7 +130,7 @@ SIG Architecture for cross-cutting KEPs). This is the title of your KEP. Keep it short, simple, and descriptive. A good title can help communicate what the KEP is and should be considered as part of - +7 any review. --> diff --git a/out b/out new file mode 100644 index 00000000000..f917f06be01 --- /dev/null +++ b/out @@ -0,0 +1 @@ +ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSghO9XxHwLdURMMMGFJC/Ew24rbqHyejKOuTDXI4knRZwLY6ckl01Ixp39/3m4LEeuh7Qpl/HjvkwETyrUUNoEGKP8+I9xapOzRyCJaG2SpU1KMsb6FhdTOCCZeSfoA2fofxArhabrn4IXfKo8rS356DgZnDZjo46+cV5fbkZtzgAa4wiEOcCyxDMuEo7wkP6BITTgOuQgqitSNZRckRoUonxV9rVpQ+PdIyW3QRw+WLyYqahoNwIEJWQnhT/DyhzKjyxG7fVWJWiaQN4j42Ly9CdKhzJ8k68UZUVjm06kPNoGB1M1KsltVOF4C+o1JCXQKKYVWeEvcQafvocijgx root@mscphis01197 From e81003a3883eb224a8186142e6d3ac25c9e43b40 Mon Sep 17 00:00:00 2001 From: Asier Gutierrez Date: Thu, 29 Dec 2022 22:37:53 +0800 Subject: [PATCH 3/6] Changed the wording in a number of points according to feedback --- .../3702-ima-namespace-support/README.md | 299 +----------------- 1 file changed, 16 insertions(+), 283 deletions(-) diff --git a/keps/sig-node/3702-ima-namespace-support/README.md b/keps/sig-node/3702-ima-namespace-support/README.md index 0496823c4f4..8afed007902 100644 --- a/keps/sig-node/3702-ima-namespace-support/README.md +++ b/keps/sig-node/3702-ima-namespace-support/README.md @@ -1,164 +1,5 @@ - - - - - # KEP-3702: IMA namespace support inside containers - - - - - - - - - - - - - [KEP-XXXXX: IMA namespace support inside containers](#kep-xxxxx-ima-namespace-support-inside-containers) - [Release Signoff Checklist](#release-signoff-checklist) - [Summary](#summary) @@ -191,46 +32,28 @@ tags, and then generate with `hack/update-toc.sh`. - - ## Release Signoff Checklist - - - - Items marked with (R) are required *prior to targeting to a milestone / release*. @@ -243,103 +66,72 @@ Items marked with (R) are required *prior to targeting to a milestone / release* [kubernetes/website]: https://git.k8s.io/website - ## Summary - +IMA namespaces allow to check the file integrity. This proposal adds regular file integrity inside containers deployed in kubernetes. -IMA namespaces allow to check the file integrity. This proposal adds file integrity inside containers deployed in kubernetes. - - - ## Motivation - - File integrity is a way to improve security in systems allowing to: * Detect illicit activity - * Detect unintended changes - * Verify the status and health of the system - * Comply with access rules - - This can be achieved using IMA (Integrity Measurement Architecture) and EVM (Extended Verification Module), which can use a TPM chip as Hardware Root of Trust for high security environments. - - ### Goals - - -* Allow IMA to work inside a container with remote attestation - - +* Check integrity of regular files inside containers, and hence check POD integrity +* Using remote attestation mechanism, check the integrity of a given POD or all PODS in the cluster deployed with IMA namespace enabled periodically +* Alert about corrupted or compromised pods ### Non-Goals - +. ## Proposal - -We propose to enable IMA linux namespaces in pods. + ### User Stories (Optional) - - #### Story 1 - As a cluster admin, I want to detect undesired file changes, so that I can take out pods that have been compromised. - +An intruder perform some malicious changes inside a certain pod's files. The system should be able to detect those changed and alert about the inconsistent pod. The remote attestation framework could keep this pod running or make a copy (for forensic analysis) and delete it. #### Story 2 - - As a cluster admin, I want to deploy a only proven and certified pods, so that I can comply with internal policies as well as security regulations. -#### Story 3 +Let's say that we are working in a high security environment where only approved images can be deployed. In this scenario we can make sure that the pod deploy used an imaged that hasn't been tampered. - +#### Story 3 As a cluster admin, I want to deny access to certain files inside the pod, so that a potential intruder can't access sensitive information. -### Notes/Constraints/Caveats (Optional) - - +In some cases, we should not even allow root to modify certain files inside the container. +### Notes/Constraints/Caveats (Optional) We need to enable IMA in the kernel and container runtimes (runC, CRI-O, docker, containerd, etc.). - ### Risks and Mitigations ## Design Details - In order for the this feature to work, the nodes where the pods will be deploy should have IMA enabled and a recent kernel that supports IMA namespaces (WIP, it should be merged soon). The nodes should also have a TPM chip. We could use nodAffinity of labels and annotations in nodes in order to select where to deploy the pods. @@ -363,29 +155,10 @@ IMA is only available in Linux hosts and Linux containers. Unfortunately, IMA is ### Runtime specification -In order to run a container we need a bundle that contains a config.json file with all the configuration. There is a Linux specific configuration section where the namespaces are listed. According to the standard, the following namespace types SHOULD be supported. - - - -* pid - -* network - -* mount - -* ipc - -* uts - -* user +There is an ongoing discussion regarding the runtime changes. -* cgroup - - - -We suggest to add a new namespaces, initially as a OPTIONAL type, in order to keep the backward compatibility. - - +https://github.com/opencontainers/runc/pull/3639 + ### CRI API @@ -436,20 +209,13 @@ Which unit tests should we include? ### Graduation Criteria - - - -This KEP is a policy KEP, not a feature KEP. It will start as GA. - #### GA - ### Upgrade / Downgrade Strategy - ### Version Skew Strategy @@ -463,68 +229,35 @@ Not applicable because this is a policy KEP. ## Implementation History - - - - ## Drawbacks - - - - ## Alternatives - - - - ## Infrastructure Needed (Optional) - - +--> \ No newline at end of file From b05513b8cd96eb4b263afd7ab74973b89824d4e5 Mon Sep 17 00:00:00 2001 From: Asier Gutierrez Date: Wed, 11 Jan 2023 21:39:24 +0800 Subject: [PATCH 4/6] Incorporated feedback from @rata --- .../3702-ima-namespace-support/README.md | 37 ++++++++++--------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/keps/sig-node/3702-ima-namespace-support/README.md b/keps/sig-node/3702-ima-namespace-support/README.md index 8afed007902..3741072fbad 100644 --- a/keps/sig-node/3702-ima-namespace-support/README.md +++ b/keps/sig-node/3702-ima-namespace-support/README.md @@ -95,9 +95,7 @@ This can be achieved using IMA (Integrity Measurement Architecture) and EVM (Ext ## Proposal ### User Stories (Optional) @@ -140,29 +138,21 @@ The linux kernel IMA namespace support is based on user namespaces. Therefore, t Should we enable IMA namespaces by default when enabling user namespaces? - There will be a CRI API change which will allow the pod to use IMA namespaces and specify the namespace policy. - - ### Linux kernel - - IMA is only available in Linux hosts and Linux containers. Unfortunately, IMA is not a separate namespace, which is needed in order to isolate it and be used inside containers. Upcoming kernel patches should add support for IMA namespaces. - ### Runtime specification There is an ongoing discussion regarding the runtime changes. -https://github.com/opencontainers/runc/pull/3639 - +https://github.com/opencontainers/runtime-spec/pull/1164 -### CRI API - +### CRI API We propose to add the following message. @@ -170,8 +160,19 @@ We propose to add the following message. ```protobuf -message NamespaceOptions { - bool ima = 6; +message LinuxSandboxSecurityContext { + NamespaceOption namespace_options = 1; + SELinuxOption selinux_options = 2; + Int64Value run_as_user = 3; + Int64Value run_as_group = 8; + bool readonly_rootfs = 4; + repeated int64 supplemental_groups = 5; + bool privileged = 6; + SecurityProfile seccomp = 9; + SecurityProfile apparmor = 10; + string seccomp_profile_path = 7 [deprecated=true]; + // new field + bool ima = 11; } ``` @@ -185,6 +186,8 @@ metadata: name: nginx spec: securityContext: + # New field + ima: true containers: - name: nginx @@ -202,14 +205,14 @@ This features will integrate with a future remote attestation procedure, which w ### Test Plan - + Which unit tests should we include? - + ### Graduation Criteria - + #### GA From c5eb082a4602dc9b458fa0bae692c714dbd38d0d Mon Sep 17 00:00:00 2001 From: Asier Gutierrez Date: Thu, 12 Jan 2023 17:57:08 +0800 Subject: [PATCH 5/6] Removed old left overs about IMA policies --- keps/sig-node/3702-ima-namespace-support/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/keps/sig-node/3702-ima-namespace-support/README.md b/keps/sig-node/3702-ima-namespace-support/README.md index 3741072fbad..14b846efdea 100644 --- a/keps/sig-node/3702-ima-namespace-support/README.md +++ b/keps/sig-node/3702-ima-namespace-support/README.md @@ -138,8 +138,6 @@ The linux kernel IMA namespace support is based on user namespaces. Therefore, t Should we enable IMA namespaces by default when enabling user namespaces? -There will be a CRI API change which will allow the pod to use IMA namespaces and specify the namespace policy. - ### Linux kernel IMA is only available in Linux hosts and Linux containers. Unfortunately, IMA is not a separate namespace, which is needed in order to isolate it and be used inside containers. Upcoming kernel patches should add support for IMA namespaces. From fbc5b7630e941af592291d2953d7eb56fe33c717 Mon Sep 17 00:00:00 2001 From: Asier Gutierrez <118451796+asierHuawei@users.noreply.github.com> Date: Thu, 12 Jan 2023 13:04:41 +0300 Subject: [PATCH 6/6] Delete out --- out | 1 - 1 file changed, 1 deletion(-) delete mode 100644 out diff --git a/out b/out deleted file mode 100644 index f917f06be01..00000000000 --- a/out +++ /dev/null @@ -1 +0,0 @@ -ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSghO9XxHwLdURMMMGFJC/Ew24rbqHyejKOuTDXI4knRZwLY6ckl01Ixp39/3m4LEeuh7Qpl/HjvkwETyrUUNoEGKP8+I9xapOzRyCJaG2SpU1KMsb6FhdTOCCZeSfoA2fofxArhabrn4IXfKo8rS356DgZnDZjo46+cV5fbkZtzgAa4wiEOcCyxDMuEo7wkP6BITTgOuQgqitSNZRckRoUonxV9rVpQ+PdIyW3QRw+WLyYqahoNwIEJWQnhT/DyhzKjyxG7fVWJWiaQN4j42Ly9CdKhzJ8k68UZUVjm06kPNoGB1M1KsltVOF4C+o1JCXQKKYVWeEvcQafvocijgx root@mscphis01197