Non existing localhostProfile Seccomp profile is not applied on Kubernetes nodes >= 1.28

[MetalPinguinInc]

What happened?

I am running 2 clusters, 1 is still on 1.26 and the other is on 1.29.5. I tried to apply a custom seccomp profile to a pod on 1.29.5 and noticed it did not seem to work. While trying to pin-point the issue, I found out that applying a custom seccomp profile that does not exist (i.e. the file does not exist on the node) to a pod on a kubernetes node below 1.28 fails to start the pod, with an error:

Error: failed to generate security options for container "test-container": failed to generate seccomp security options for container: cannot load seccomp profile "/var/lib/kubelet/seccomp/profiles/audit.json": open /var/lib/kubelet/seccomp/profiles/audit.json: no such file or directory

However, when trying to do the same on a kubernetes node >= 1.28 incorrectly starts the pod, with no mention of the seccomp profile file not being found or being invalid. Although I used minikube to reproduce this bug, I observed it first on a bare metal installation.

What did you expect to happen?

I expect a pod not to start when the seccomp profile cannot be loaded.

How can we reproduce it (as minimally and precisely as possible)?

minikube start --kubernetes-version 1.27
kubectl apply -f audit-pod.yaml

Observe that the pod does not start and gives an error:

Error: failed to generate security options for container "test-container": failed to generate seccomp security options for container: cannot load seccomp profile "/var/lib/kubelet/seccomp/profiles/audit.json": open /var/lib/kubelet/seccomp/profiles/audit.json: no such file or directory

minikube delete
minikube start --kubernetes-version 1.28\
kubectl apply -f audit-pod.yaml\

Observe that the pod has started without any errors.

audit-pod.yaml:

kind: Pod
metadata:
  name: audit-pod
  labels:
    app: audit-pod
spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/audit.json
  containers:
  - name: test-container
    image: hashicorp/http-echo:1.0
    args:
    - "-text=just made some syscalls!"
    securityContext:
      allowPrivilegeEscalation: false

Anything else we need to know?

No response

Kubernetes version

Version that errors on invalid seccomp profile:

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.7
WARNING: version difference between client (1.29) and server (1.27) exceeds the supported minor version skew of +/-1

Version that does not error:

$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Cloud provider

None

OS version

# On Linux:
$ cat /etc/os-release
NAME=Buildroot
VERSION=2021.02.12-1-gb75713b-dirty
ID=buildroot
VERSION_ID=2021.02.12
PRETTY_NAME="Buildroot 2021.02.12"
$ uname -a
Linux minikube 5.10.57 #1 SMP Tue Nov 7 06:51:54 UTC 2023 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[neolit123]

/sig node security

[chengjoey]

related pr #117050

issue #117045

[chengjoey]

/assign

[chengjoey]

hi @MetalPinguinInc , if you use containerd in k8s >= 1.28, you will still see error message:

cannot load seccomp profile

use contaienrd as coantienr runtime

minikube start --kubernetes-version=1.28.1 --container-runtime=containerd

I guess this is caused by different container runtimes, and your cri should be docker

[MetalPinguinInc]

hi @chengjoey on my own baremetal clusters I am indeed running Docker with containerd as the container runtime and cri-dockerd as a shim between Kubernetes and Docker.

I can indeed confirm that using containerd as the runtime directly, seems to correctly throw the error in minikube. This has left me confused. Without using the --container-runtime=containerd flag, both minikube start --kubernetes-version=1.28.1 and minikube start --kubernetes-version=1.27 report exactly the same docker versions:

> docker version
Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:05:28 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.8
  GitCommit:        8e4b0bde866788eec76735cc77c4720144248fb7
 runc:
  Version:          1.1.9
  GitCommit:        ccaecfcbc907d70a7aa870a6650887b901b25b82
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I am slightly confused by the terminology here: Kubernetes can run on both Docker (in which case you also need cri-dockerd) and Containerd, but Docker uses containerd as its container runtime anyway. When I started using Kubernetes only Docker was used, is there any reason to still have docker in the mix or is this mostly still supported for backwards compatibility and is using containerd with kubernetes directly a more streamlined approach?

In anycase, could it be that the issue lies in how Kubernetes communicates security options to Docker in versions >= 1.28? Since the Docker version is the same between 1.28 and 1.27 it seems unlikely that this is a issue in Docker.

[haircommander]

It sounds like this is an issue with cri-dockerd, not kubernetes. I'd be curious how cri-o handles it, but if you wish to keep using cri-dockerd and want this situation fixed, I recommend opening an issue with them.

/close

please reopen if you think there's something wrong with kubernetes :)

[k8s-ci-robot]

@haircommander: Closing this issue.

In response to this:

It sounds like this is an issue with cri-dockerd, not kubernetes. I'd be curious how cri-o handles it, but if you wish to keep using cri-dockerd and want this situation fixed, I recommend opening an issue with them.

/close

please reopen if you think there's something wrong with kubernetes :)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.