Overview
Container security can be categorized as follows:
- 1. Whether there is a virus in the container (+ image)—representing the project: clair, IBM Vulnerability Advisor
- 2. Whether the container (+ image) has been tampered with - on behalf of the project: Notary
- 3. Whether the container is isolated (container runtime security) - represents the project: runc
- 4. Whether the container (+ image) is kept secret, that is, whether the image is run by the target user—representing the project: containerd, cri-o
This article mainly discusses the fourth point - image confidentiality. The community's current work on image confidentiality is largely led by IBM's Brandon Lum and Harshal Patil Dominate. Due to image encryption characteristics will change OCI standard Image format section in: the Image Specification image-spec , so it is necessary to understand OCI before introducing the principle of image encryption in detail
OCI
The OCI(Open Container Initiative) was launched in June 2015 by Docker, CoreOS, and other leaders in the container industry. It is committed to creating open industry standards around container formats and runtimes. It mainly consists of two parts:
- 1. the Runtime Specification (runtime-spec) - responsible for the container runtime, contains technologies such as namespaces, cgroups, Linux capabilities, as well as SELinux, AppArmor, and Seccomp profiles
- 2. the Image Specification (image-spec) - responsible for the container image standard, mainly standardizing the image storage format
The community's implementation of the OCI specification is as follows:
- OCI Runtime Spec Implementations: runc(Runtime (Container)), hyperhq/runv(Runtime (Virtual Machine))
- OCI Image Spec Implementations: Buildah (build OCI image), Skopeo (Transfer OCI image, based on containers/image)
The items supporting OCI runtime spec&image spec are as follows:
- containerd :exist runc Based on the realization of the OCI Image Spec part
- cri-o :exist runc Based on the realization of the OCI Image Spec part (base on containers/image, containers/storage and CNI)
- Podman :exist Buildah Based on it, OCI image&runtime spec is implemented( Buildah and Podman relationship)
- docker : Wraps the docker API on the basis of containerd, and plans to migrate the underlying related code to containerd(moby/moby#38043)
- rkt(pronounced like a "rocket")
For this reason, containerd, cri-o, docker and rkt are also broadly referred to as container runtime(High-Level Container Runtimes), as shown in the figure:
For details, see An Introduction to Container Runtimes
Encrypted Container Image Principle
Since image encryption is an extension of the OCI image format, we first introduce the existing image format, as follows:
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"size": 7023,
"digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 32654,
"digest": "sha256:9834876dcfb05cb167a5c24953eba58c4ac89b1adf57f28f2f9d09af107ee8f0"
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 16724,
"digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 73109,
"digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
}
],
"annotations": {
"com.example.key1": "value1",
"com.example.key2": "value2"
}
}
Unlike the image index, which contains information about a set of images that can span a variety of architectures and operating systems, an image manifest provides a configuration and set of layers for a single container image for a specific architecture and operating system.
The format is described as follows (refers to Image Manifest Property Descriptions):
- schemaVersion: This REQUIRED property specifies the image manifest schema version
- config(belongs to descriptor ): contains the metafile (configuration file) information of the image
- mediaType: identifies the config file format (media type of the referenced content), the format that must be implemented at present is application/vnd.oci.image.config.v1+json
- digest: sha256 hash value of the content of the config file
- size: config file content size (in bytes)
- layers: Contains information about each layer of the mirror, the content is the layered by the mirror (belongs to descriptor ) an array of
- mediaType: identifies the image layer file format (media type of the referenced content). Currently, the list of formats that must be implemented is as follows:
- digest: sha256 hash value of the content of the layer file
- size: layer file content size (in bytes)
- annotations(string-string map): This OPTIONAL property contains arbitrary metadata for the image manifest.
and based on Proposal: New Mediatype - Container Image Encryption The discussion of the image encryption schematic (draft of modifications to the OCI spec on Encrypted Container Images) is as follows:
Encrypted container images are based on the OCI image spec. The changes to the spec is the adding of the encrypted layer mediatype. An image manifest contains some metadata and a list of layers. We introduce a new layer mediatype suffix +encrypted to represent an encrypted layer. For example, a regular layer with mediatype 'application/vnd.oci.image.layer.v1.tar' when encrypted would be 'application/vnd.oci.image.layer.v1.tar+encrypted'. Because there is some metadata involved with the encryption, an encrypted layer will also contain several annotations with the prefix org.opencontainers.image.enc.
That is, a mediaType type is added on the basis of the original format, indicating that the data file is encrypted; at the same time, specific encryption-related information is added to the annotation.
For example, the data before encryption is as follows:
"layers":[
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
"digest":"sha256:7c9d20b9b6cda1c58bc4f9d6c401386786f584437abbe87e58910f8a9a15386b",
"size":760770
}
]
The encrypted data is as follows:
"layers":[
{
"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
"digest":"sha256:c72c69b36a886c268e0d7382a7c6d885271b6f0030ff022fda2b6346b2b274ba",
"size":760770,
"annotations": {
"org.opencontainers.image.enc.keys.jwe":"eyJwcm90ZWN0Z...",
"org.opencontainers.image.enc.pubopts":"eyJjaXBoZXIiOi..."
}
}
]
In addition, by refining the encryption granularity to the image layer, the layering and deduplication characteristics of the original image format are preserved
The entire encryption system uses Hybrid encryption scheme:
- 1. Use symmetric encryption algorithm (i.e. AES) to encrypt image layered data - mainly to improve encryption and decryption speed
- 2. Use an asymmetric encryption algorithm (eg: OpenPGP, JSON Web Encryption (JWE), and PKCS#7) to first encrypt the above symmetric encryption key with the public key, and then use the private key to decrypt the encrypted key—— Mainly to improve the flexibility of key management
The flow chart is as follows:
It can be roughly summarized into the following steps:
- 1. Generate a symmetric encryption key (i.e. LEK) for encrypting image layering
- 2. Encrypt LEK with asymmetric encryption public key to generate Wrapped Key
- 3. Upload the encrypted image (including Wrapped Key) to a public repository, for example: Docker Hub or quay.io
- 4. The holder of the asymmetric encryption private key downloads the image, and uses the private key to decrypt the image and run it
Encrypted Container Image Build ToolChain-containerd imgcrypt
For image encryption tools, the community plans to support: docker CLI(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714), containerd imgcrypt, skopeo as well as buildah , containerd imgcrypt and skopeo have been implemented so far
Below we are based on containerd imgcrypt Practice the above encryption process:
# Step1: Install containerd imgcrypt
$ git clone https://github.com/containerd/imgcrypt.git github.com/containerd/imgcrypt
$ cd github.com/containerd/imgcrypt && make && make install
install
# Step2: Setting up containerd
$ wget https://github.com/containerd/containerd/releases/download/v1.3.0/containerd-1.3.0.linux-amd64.tar.gz
$ tar -xzf containerd-1.3.0.linux-amd64.tar.gz
$ ls bin/
containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress ctr
$ cat <<EOF > config.toml
disable_plugins = ["cri"]
root = "/tmp/var/lib/containerd"
state = "/tmp/run/containerd"
[grpc]
address = "/tmp/run/containerd/mycontainerd.sock"
uid = 0
gid = 0
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
path = "/usr/local/bin/ctd-decoder"
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
returns = "application/vnd.oci.image.layer.v1.tar"
path = "/usr/local/bin/ctd-decoder"
EOF
$ bin/containerd -c config.toml
[... truncated ...]
INFO[2020-02-26T11:16:07.704084989+08:00] containerd successfully booted in 0.047800s
# Step3: Generate some RSA keys with openssl
$ openssl genrsa -out mykey.pem
Generating RSA private key, 2048 bit long modulus
.......................................................................................................+++
.............................+++
e is 65537 (0x10001)
$ openssl rsa -in mykey.pem -pubout -out mypubkey.pem
writing RSA key
# Step4: Pulling an image
$ chmod 0666 /tmp/run/containerd/containerd.sock
$ CTR="/usr/local/bin/ctr-enc -a /tmp/run/containerd/containerd.sock"
$ $CTR images pull --all-platforms docker.io/library/bash:latest
[... truncated ...]
elapsed: 8.7 s total: 39.2 M (4.5 MiB/s)
unpacking linux/amd64 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/arm/v6 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/arm/v7 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/arm64/v8 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/386 sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/ppc64le sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
unpacking linux/s390x sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f...
done
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
$ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
# Step5: Encrypting the image
$ $CTR images encrypt --recipient jwe:mypubkey.pem --platform linux/amd64 docker.io/library/bash:latest bash.enc:latest
Encrypting docker.io/library/bash:latest to bash.enc:latest
# The arguments are:
# --recipient jwe:mypubkey.pem: This indicates that we want to encrypt the image using the public key mypubkey.pem that we just generated, and the prefix jwe: indicates that we want to use the encryption scheme JSON web encryption scheme for our encryption metadata.
# --platform linux/amd64: Encrypt only the linux/amd64 image
# docker.io/library/bash:latest - The image to encrypt
# bash.enc:latest - The tag of the encrypted image to be created
# Optional: it is possible to encrypt just certain layers of the image by using the --layer flag
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
# We can observe the layerinfo to show the encryption
$ $CTR images layerinfo --platform linux/amd64 bash.enc:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74 linux/amd64 2802957 jwe [jwe]
1 sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d linux/amd64 3186680 jwe [jwe]
2 sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87 linux/amd64 340 jwe [jwe]
# Strange enough, the layers 'SIZE' of 'bash.enc:latest' is same with the one of 'docker.io/library/bash:latest'
$ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
# Compare other platforms
$ $CTR images layerinfo --platform linux/arm64/v8 bash.enc:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:8fa90b21c985a6fcfff966bdfbde81cdd088de0aa8af38110057f6ac408f4408 linux/arm64/v8 2723075
1 sha256:4d89eb6b628bc985ffcdcb939bf05a16194be6aed18c2848e1424d01ca0012fd linux/arm64/v8 3193678
2 sha256:4818fb358d8f64c071010cb05fe59285888a36edbe1a45a5eed80576b08d312c linux/arm64/v8 344
$ $CTR images layerinfo --platform linux/arm64/v8 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:8fa90b21c985a6fcfff966bdfbde81cdd088de0aa8af38110057f6ac408f4408 linux/arm64/v8 2723075
1 sha256:4d89eb6b628bc985ffcdcb939bf05a16194be6aed18c2848e1424d01ca0012fd linux/arm64/v8 3193678
2 sha256:4818fb358d8f64c071010cb05fe59285888a36edbe1a45a5eed80576b08d312c linux/arm64/v8 344
# Step6: Pushing to registry
# Let us set up a local registry that we can push the encrypted image to. Note that by default, the docker/distribution registry version 2.7.1 and above supports encrypted OCI images out of the box.
$ docker run -d -p 5000:5000 --restart=always --name registry registry:2.7.1
Unable to find image 'registry:2.7.1' locally
2.7.1: Pulling from library/registry
486039affc0a: Pull complete
ba51a3b098e6: Pull complete
8bb4c43d6c8e: Pull complete
6f5f453e5f2d: Pull complete
42bc10b72f42: Pull complete
Digest: sha256:7d081088e4bfd632a88e3f3bcd9e007ef44a796fddfe3261407a3f9f04abe1e7
Status: Downloaded newer image for registry:2.7.1
f65eff4accdc64eca42b5aa2c2c36e831c63172472800c835e031ec758a3ccbe
$ docker ps |grep registry
f65eff4accdc registry:2.7.1 "/entrypoint.sh /etc..." 28 seconds ago Up 27 seconds 0.0.0.0:5000->5000/tcp registry
$ $CTR images tag bash.enc:latest localhost:5000/bash.enc:latest
localhost:5000/bash.enc:latest
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
$ $CTR images push localhost:5000/bash.enc:latest
[... truncated ...]
elapsed: 0.4 s total: 39.2 M (97.7 MiB/s)
$ $CTR images rm localhost:5000/bash.enc:latest bash.enc:latest
localhost:5000/bash.enc:latest
bash.enc:latest
$ $CTR images pull localhost:5000/bash.enc:latest
localhost:5000/bash.enc:latest: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:043e2439dae1a9634be8b3a589c8a2e6048f0f1c0ca9d1516c53cd1ee8a8bdf8: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:e155078e7721d0d550b869592482f0e2c9b9c40e6a541fb430ab46a278fde5cd: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.2 s total: 3.0 Mi (15.1 MiB/s)
unpacking linux/amd64 sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa...
done
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
# Step7: Decrypting the image
$ $CTR images decrypt --key mykey.pem --platform linux/amd64 localhost:5000/bash.enc:latest localhost:5000/bash.dec:latest
Decrypting localhost:5000/bash.enc:latest to localhost:5000/bash.dec:latest
$ $CTR images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/bash:latest application/vnd.docker.distribution.manifest.list.v2+json sha256:8193ca23afc8e1069bd3d982733df79c68dbcfcd66b4da6b5d65da85987dae2f 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.dec:latest application/vnd.oci.image.index.v1+json sha256:52dbac1ba1464568b2de8c9bc0d1b08d26786365e415c2c3ccb52a23191afc22 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
localhost:5000/bash.enc:latest application/vnd.oci.image.index.v1+json sha256:e1cea4c43a0d43bf3fdd6af66509668f6fe16e433085fcc7cf5b8a4ec48c80aa 5.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x -
# Notice that the 'DIGEST' of 'localhost:5000/bash.dec:latest' is different with the one of 'docker.io/library/bash:latest'
# Let's have a look at layerinfos of these images
$ $CTR images layerinfo --platform linux/amd64 docker.io/library/bash:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
$ $CTR images layerinfo --platform linux/amd64 localhost:5000/bash.enc:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:375abe9071e498e57f1cfcf349266948c3760bfb136e051c22a532d38a020e74 linux/amd64 2802957 jwe [jwe]
1 sha256:3298803dec193fdd561ea27d0b1469e3f0b5b796962f543f3bcc57cd12ebdf5d linux/amd64 3186680 jwe [jwe]
2 sha256:bdbc04a60398d706a06e5ae88a4cdbfef770c9fe1b12e31fcf5a50532dafad87 linux/amd64 340 jwe [jwe]
$ $CTR images layerinfo --platform linux/amd64 localhost:5000/bash.dec:latest
# DIGEST PLATFORM SIZE ENCRYPTION RECIPIENTS
0 sha256:c9b1b535fdd91a9855fb7f82348177e5f019329a58c53c47272962dd60f71fc9 linux/amd64 2802957
1 sha256:a3698fd3137d820dbb91b5f96e89af64b196dde4ab4c326dd1bd6291bbd771cf linux/amd64 3186680
2 sha256:2538f4b330c99e67e96803569826ba62f7a3353d27f2a15bcf1e33d1814fa7ad linux/amd64 340
# layers of 'localhost:5000/bash.dec:latest' is same with 'docker.io/library/bash:latest' but the image 'DIGEST' differs
# Step8: Running a container
# We can now attempt to run the encrypted container image
$ $CTR run --rm localhost:5000/bash.enc:latest test echo 'Hello World!'
ctr: you are not authorized to use this image: missing private key needed for decryption
# We notice that running the image failed. This is because we did not provide the keys for the encrypted image.We can pass the keys in with the --key flag.
$ $CTR run --rm --key mykey.pem localhost:5000/bash.enc:latest test echo 'Hello World!'
Hello World!
Kubernetes Image Encryption Support
Currently the community plans to support Kubernetes Two image encryption modes:
- Node Key Model: Put the key on the Kubernetes worker node, with the node as the decryption granularity (implemented)
- Multitenant Key Model: Multitenant model, which supports decryption granularity for clusters and users (not yet implemented, based on this KEP)
This article mainly discusses the first model, as shown in the figure:
The principle is very clear:
- 1. Generate the encryption public key and private key, and put the key in the specified path of the Worker node of the Kubernetes cluster
- 2. Encrypt the image with the public key and upload it to the mirror warehouse
- 3. Create a service in a Kubernetes cluster with an encrypted image
- 4. The Worker node Kubelet will call the container runtime to pull the encrypted image, use the private key to decrypt the image, and finally run the decrypted image successfully
After understanding the general process, let's further discuss the support of the container runtime on the Worker node for image encryption
First, Kubernetes uses CRI To dock the container runtime, as shown in the figure:
For the image encryption feature, the community plans to support container-runtime: containerd(+CRI-Containerd )as well as cri-o , the currently implemented containerd( CRI-Containerd is currently merging a PR ) and cri-o. Therefore, here we mainly discuss container runtime implementations that plan to support image encryption features such as containerd and cri-o
Here is an illustration of the use of cri combined with cri-o, containerd and docker:
Through the above diagram, we can more intuitively see the details of the use of the above container runtime by Kubernetes
Kubernetes Image Encryption Support-CRIO
This chapter gives an example of using Kubernetes combined with CRIO based on the Node Key Model image encryption mode:
# Step1: Install cri-o from source code
# Install Runtime dependencies
$ yum install -y \
containers-common \
device-mapper-devel \
git \
glib2-devel \
glibc-devel \
glibc-static \
go \
gpgme-devel \
libassuan-devel \
libgpg-error-devel \
libseccomp-devel \
libselinux-devel \
pkgconfig \
make \
runc
# Compile cri-o from source code
$ git clone https://github.com/cri-o/cri-o go/src/github.com/cri-o/cri-o
$ cd go/src/github.com/cri-o/cri-o && make
mkdir -p "/root/go/src/github.com/cri-o/cri-o/_output/src/github.com/cri-o"
ln -s "/root/go/src/github.com/cri-o/cri-o" "/root/go/src/github.com/cri-o/cri-o/_output/src/github.com/cri-o/cri-o"
touch "/root/go/src/github.com/cri-o/cri-o/_output/.gopathok"
GO111MODULE=on go build --mod=vendor -ldflags '-s -w -X main.gitCommit="186c23056ac25396e9ea9e49c5e5bfbe271b7751" -X main.buildInfo=1582724503' -tags "containers_image_ostree_stub exclude_graphdriver_btrfs btrfs_noversion seccomp selinux" -o bin/crio github.com/cri-o/cri-o/cmd/crio
GO111MODULE=on go build --mod=vendor -ldflags '-s -w -X main.gitCommit="186c23056ac25396e9ea9e49c5e5bfbe271b7751" -X main.buildInfo=1582724526' -tags "containers_image_ostree_stub exclude_graphdriver_btrfs btrfs_noversion seccomp selinux" -o bin/crio-status github.com/cri-o/cri-o/cmd/crio-status
make -C pinns
make[1]: Entering directory `/root/go/src/github.com/cri-o/cri-o/pinns'
cc -std=c99 -Os -Wall -Wextra -static -c -o pinns.o pinns.c
cc -o ../bin/pinns pinns.o -std=c99 -Os -Wall -Wextra -static
make[1]: Leaving directory `/root/go/src/github.com/cri-o/cri-o/pinns'
./bin/crio --config="" config > crio.conf
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio-status.8.md -out docs/crio-status.8.tmp && touch docs/crio-status.8.tmp && mv docs/crio-status.8.tmp docs/crio-status.8) || \
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio-status.8.md -out docs/crio-status.8.tmp && touch docs/crio-status.8.tmp && mv docs/crio-status.8.tmp docs/crio-status.8)
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.conf.5.md -out docs/crio.conf.5.tmp && touch docs/crio.conf.5.tmp && mv docs/crio.conf.5.tmp docs/crio.conf.5) || \
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.conf.5.md -out docs/crio.conf.5.tmp && touch docs/crio.conf.5.tmp && mv docs/crio.conf.5.tmp docs/crio.conf.5)
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.8.md -out docs/crio.8.tmp && touch docs/crio.8.tmp && mv docs/crio.8.tmp docs/crio.8) || \
(/root/go/src/github.com/cri-o/cri-o/build/bin/go-md2man -in docs/crio.8.md -out docs/crio.8.tmp && touch docs/crio.8.tmp && mv docs/crio.8.tmp docs/crio.8)
$ make install
install -D -m 755 bin/crio /usr/local/bin/crio
install -D -m 755 bin/crio-status /usr/local/bin/crio-status
install -D -m 755 bin/pinns /usr/local/bin/pinns
install -d -m 755 /usr/local/share/man/man5
install -d -m 755 /usr/local/share/man/man8
install -m 644 docs/crio.conf.5 -t /usr/local/share/man/man5
install -m 644 docs/crio-status.8 docs/crio.8 -t /usr/local/share/man/man8
install -d -m 755 /usr/local/share/bash-completion/completions
install -d -m 755 /usr/local/share/fish/completions
install -d -m 755 /usr/local/share/zsh/site-functions
install -D -m 644 -t /usr/local/share/bash-completion/completions completions/bash/crio
install -D -m 644 -t /usr/local/share/fish/completions completions/fish/crio.fish
install -D -m 644 -t /usr/local/share/zsh/site-functions completions/zsh/_crio
install -D -m 644 -t /usr/local/share/bash-completion/completions completions/bash/crio-status
install -D -m 644 -t /usr/local/share/fish/completions completions/fish/crio-status.fish
install -D -m 644 -t /usr/local/share/zsh/site-functions completions/zsh/_crio-status
# Download conmon
$ git clone https://github.com/containers/conmon ~/go/src/github.com/containers/conmon
$ cd ~/go/src/github.com/containers/conmon
$ make
mkdir -p bin
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/conmon.o -c src/conmon.c
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/cmsg.o -c src/cmsg.c
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/ctr_logging.o -c src/ctr_logging.c
cc -std=c99 -Os -Wall -Wextra -Werror -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DVERSION=\"2.0.11-dev\" -DGIT_COMMIT=\""86aa80b908d7532ab9b6edd6f4f27ba4bf6ba17b"\" -D USE_JOURNALD=0 -o src/utils.o -c src/utils.c
cc -o bin/conmon src/conmon.o src/cmsg.o src/ctr_logging.o src/utils.o -lglib-2.0 -lsystemd
$ make install
install -D -m 755 bin/conmon /usr/local/bin/conmon
# Setup CNI networking
# Set CNI network configurations
$ cd /root/go/src/github.com/cri-o/cri-o && mkdir -p /etc/cni/net.d
$ cp contrib/cni/*.conf /etc/cni/net.d/
$ ls /etc/cni/net.d/
10-crio-bridge.conf 99-loopback.conf
# Install the CNI plugins
$ git clone https://github.com/containernetworking/plugins ~/go/src/github.com/containernetworking/plugins
$ cd ~/go/src/github.com/containernetworking/plugins && git checkout v0.8.5
$ ./build_linux.sh
Building plugins
bandwidth
firewall
flannel
portmap
sbr
tuning
bridge
host-device
ipvlan
loopback
macvlan
ptp
vlan
dhcp
host-local
static
$ mkdir -p /opt/cni/bin
$ cp bin/* /opt/cni/bin/
# Generating CRI-O configuration
$ cd ~/go/src/github.com/cri-o/cri-o/ && make install.config
install -d /usr/local/share/containers/oci/hooks.d
install -D -m 644 crio.conf /etc/crio/crio.conf
install -D -m 644 crio-umount.conf /usr/local/share/oci-umount/oci-umount.d/crio-umount.conf
install -D -m 644 crictl.yaml /etc
# Path to OCI hooks directories for automatically executed hooks.
hooks_dir = [
"/usr/local/share/containers/oci/hooks.d",
]
# Validate registries in registries.conf
# Edit /etc/containers/registries.conf and verify that the registries option has valid values in it.
$ cat /etc/containers/registries.conf
[... truncated ...]
[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', 'registry.centos.org']
[registries.insecure]
registries = []
[registries.block]
registries = []
# Recommended - Use systemd cgroups.
# By default, CRI-O uses cgroupfs as a cgroup manager. However, we recommend using systemd as a cgroup manager. You can change your cgroup manager in crio.conf:
cgroup_manager = "systemd"
# Optional - Modify verbosity of logs in /etc/crio/crio.conf
# Users can modify the log_level field in /etc/crio/crio.conf to change the verbosity of the logs. Options are fatal, panic, error (default), warn, info, and debug.
log_level = "info"
# Starting CRI-O
$ make install.systemd
install -D -m 644 contrib/systemd/crio.service /usr/local/lib/systemd/system/crio.service
ln -sf crio.service /usr/local/lib/systemd/system/cri-o.service
install -D -m 644 contrib/systemd/crio-shutdown.service /usr/local/lib/systemd/system/crio-shutdown.service
install -D -m 644 contrib/systemd/crio-wipe.service /usr/local/lib/systemd/system/crio-wipe.service
$ systemctl daemon-reload
$ systemctl enable crio
Created symlink from /etc/systemd/system/multi-user.target.wants/crio.service to /usr/local/lib/systemd/system/crio.service.
# enable crio proxy(pulling image)
$ cat /usr/local/lib/systemd/system/crio.service
[... truncated ...]
[Service]
Environment="GOTRACEBACK=crash" "HTTP_PROXY=x.x.x.x" "HTTPS_PROXY=x.x.x.x" "NO_PROXY=localhost,127.0.0.1"
$ systemctl start crio
$ systemctl status crio
* crio.service - Container Runtime Interface for OCI (CRI-O)
Loaded: loaded (/usr/local/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-02-26 22:27:38 CST; 1min 11s ago
Docs: https://github.com/cri-o/cri-o
Main PID: 16267 (crio)
CGroup: /system.slice/crio.service
`-16267 /usr/local/bin/crio
Feb 26 22:27:37 vm-xxx-centos systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855228868+08:00" level=warning msg="Not using nat...o fix"
Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855645605+08:00" level=info msg="Found CNI networ....conf"
Feb 26 22:27:37 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:37.855703783+08:00" level=info msg="Found CNI networ....conf"
Feb 26 22:27:38 vm-xxx-centos crio[16267]: W0226 22:27:38.114732 16267 hostport_manager.go:69] The binary conntrack ...eanup.
Feb 26 22:27:38 vm-xxx-centos crio[16267]: time="2020-02-26 22:27:38.114771405+08:00" level=info msg="no seccomp profi...fault"
Feb 26 22:27:38 vm-xxx-centos systemd[1]: Started Container Runtime Interface for OCI (CRI-O).
# Step2: Install crictl
$ VERSION="v1.17.0"
$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
$ tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
crictl
$ rm -f crictl-$VERSION-linux-amd64.tar.gz
# To ensure that we've setup CRI-O correctly, we will install crictl to verify our setup.
$ crictl -r unix:///run/crio/crio.sock info
{
"status": {
"conditions": [
{
"type": "RuntimeReady",
"status": true,
"reason": "",
"message": ""
},
{
"type": "NetworkReady",
"status": true,
"reason": "",
"message": ""
}
]
}
}
# Step3: Setting up a single node k8s cluster
$ wget https://github.com/kubernetes/kubernetes/archive/v1.17.1.tar.gz
$ tar -xzf v1.17.1.tar.gz && cd kubernetes-1.17.1
$ hack/install-etcd.sh
Downloading https://github.com/coreos/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz succeed
etcd v3.4.3 installed. To use:
export PATH="/root/go/src/github.com/kubernetes-1.17.1/third_party/etcd:${PATH}"
$ export PATH="/root/go/src/github.com/kubernetes-1.17.1/third_party/etcd:${PATH}"
$ CGROUP_DRIVER=systemd \
CONTAINER_RUNTIME=remote \
CONTAINER_RUNTIME_ENDPOINT='unix:///var/run/crio/crio.sock' \
./hack/local-up-cluster.sh
[... truncated ...]
To start using your cluster, you can open up another terminal/tab and run:
export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
cluster/kubectl.sh
Alternatively, you can write to the default kubeconfig:
export KUBERNETES_PROVIDER=local
cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
cluster/kubectl.sh config set-context local --cluster=local --user=myself
cluster/kubectl.sh config use-context local
cluster/kubectl.sh
$ cluster/kubectl.sh get nodes
NAME STATUS ROLES AGE VERSION
127.0.0.1 Ready <none> 21m v1.17.1
# Step4: Generating encrypted image with skopeo
# Install skopeo
$ yum install libgpgme-devel device-mapper-devel libbtrfs-devel glib2-devel libassuan-devel
$ git clone https://github.com/containers/skopeo $GOPATH/src/github.com/containers/skopeo
$ cd $GOPATH/src/github.com/containers/skopeo && make binary-local
$ make install
$ mkdir /etc/containers
$ cp default-policy.json /etc/containers/policy.json
# Pull image
$ skopeo copy docker://docker.io/library/nginx:1.15 oci:local_nginx:1.15
Getting image source signatures
Copying blob 743f2d6c1f65 done
Copying blob 6bfc4ec4420a done
Copying blob 688a776db95f done
Copying config 0fb15759be done
Writing manifest to image destination
Storing signatures
# Encrypt image
$ skopeo copy --encryption-key jwe:./mypubkey.pem oci:local_nginx:1.15 oci:nginx_encrypted:1.15
Getting image source signatures
Copying blob 743f2d6c1f65 done
Copying blob 6bfc4ec4420a done
Copying blob 688a776db95f done
Copying config 0fb15759be done
Writing manifest to image destination
Storing signatures
# Push image
$ skopeo copy --dest-tls-verify=false oci:nginx_encrypted:1.15 docker://x.x.x.x/nginx_encrypted:1.15
Getting image source signatures
Copying blob cd2a4504255e done
Copying blob 8df8b398a84d done
Copying blob 663a05dac0ac done
Copying config 0fb15759be done
Writing manifest to image destination
Storing signatures
# Step5: Getting and running an encrypted image
$ cat <<EOF > enc-dply.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: enc-nginx
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: x.x.x.x/nginx_encrypted:1.15
ports:
- containerPort: 80
EOF
$ cluster/kubectl.sh create -f enc-dply.yaml
deployment.apps/enc-nginx created
$ cluster/kubectl.sh get pods
NAME READY STATUS RESTARTS AGE
enc-nginx-7fb4578896-9qhtr 0/1 ImagePullBackOff 0 12s
enc-nginx-7fb4578896-cdrdq 0/1 ErrImagePull 0 12s
enc-nginx-7fb4578896-j94rj 0/1 ErrImagePull 0 12s
# We notice that our pods failed to run due to failure in the image pull process. Let's describe the pod to find out more.
$ cluster/kubectl.sh describe pods/enc-nginx-7fb4578896-9qhtr
[... truncated ...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/enc-nginx-7fb4578896-9qhtr to 127.0.0.1
Normal Pulling 25s (x2 over 36s) kubelet, 127.0.0.1 Pulling image "x.x.x.x/nginx_encrypted:1.15"
Warning Failed 25s (x2 over 36s) kubelet, 127.0.0.1 Failed to pull image "x.x.x.x/nginx_encrypted:1.15": rpc error: code = Unknown desc = Error decrypting layer sha256:cd2a4504255eaf69fb1ee6c5961625854cd69597a1c5722e86062fb4ab688cf8: missing private key needed for decryption
Warning Failed 25s (x2 over 36s) kubelet, 127.0.0.1 Error: ErrImagePull
Normal BackOff 13s (x2 over 36s) kubelet, 127.0.0.1 Back-off pulling image "x.x.x.x/nginx_encrypted:1.15"
Warning Failed 13s (x2 over 36s) kubelet, 127.0.0.1 Error: ImagePullBackOff
# Configuring decryption keys for CRI-O
$ mkdir -p /etc/crio/keys
$ cat << EOF > /etc/crio/keys/mykey.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA3B07uvDP+NxYXYQxjknPml1zSacijqoj1D79VzSUjjMM6th/
doKJbeMTybehF9PDly6TleluZWZiXMtqagmeEI/sUKaaq0yrMrNrzfIfJYCD0hJx
8tBFZKwUVKaYQjB4Bo6Ij0V04dcP8m6NAfXNG0/MB56zj7WhMndOPP/tuIGmj1jV
jBopq/Pm8KUrzDVCmD+kQLT7hZTi0lapG7A/tE7bc2pGypP2hLR0i9/ckieXh4IL
YKdaNda4THjnA3bHC7/ELOWPinrmQlaxY1mW8hkLCHCQ/MG/SJKkVTY2fX62BscY
KGQPOwh9jNWIS8MAbNJafYBxVueje6Xe/e+OdwIDAQABAoIBAQDXOH4+u1eerVR5
m9gYmHM1LEqdqZ5QgGuoDC8KJY9buu7WcfmvltNpbq7afYI2GgkUuaX03tniq8lh
kkPqipzS9ObLtRtmgwCiAm1WYXey44YA0ag5EwvG87qtSnd1wI6bWqKL9A3lBLPD
B/U4BW8XVV7Z1IMd8So8fgsx+cwmqk2+CqMZsq2m7FBjrCt7fu0QMaOXvuUWka5D
K8F+yvAbD5UMXOqN9fjMuPzbtiOFXvmWtpC3L6wxZjmAPuiR9fNzwM4jCXSasMU8
PNy3x0Esn/ic7IP+1v1PEiXImLui4WPF74v8i6HB6FgxCBht6rCYyl8kYh68gJS6
HTmgf9+BAoGBAPk+ft+R4T/iGDZ0y1YhRNN5zxqtOEcLf1+5/s6W6b+FTRFrxYLL
A8ZMkn0Se8DfVjPald1gMSikE8vj8YJSHAMlstPiHiYQKsBMGbjksZLjBpSapkTy
iMjpk+SmVN3sZRU3jfrcMoPfAQx0hBrCUJdhZFV/aYLiuJ7GUub34gKzAoGBAOIU
mu5aht3MDjiR2mnNzEj2TY7YED7HNWOQJ0FzqLfl4jZTowMbnz8aFUWxkJmC3i1y
0Y8rWbxV7CwcXc9fOR8jRNx2Jn6eaA4A9+a02D1hNHJ6bQivy6yloQXCs4/wardY
EBHvepygs+v507khX146bl8PmUCIYpjVFaeOrRctAoGAehhLPmnP1eODyOld0ktp
086PzZmdP/A57ULHt5vl1ZQPNMF+d5vLtZA9ElfDl6/QIoapc1BzxFzb9b0ryZM/
das59uGFs0+oIZsl3pTpB/N+fb1kRdIpf4IsmI2CdVQgEEyumHzVohPUB63sKM+X
exCSfe90WFGH7v9oDQzRAlECgYBocSRx4JhVdqNLNvYz0sMBIegKiX5XwifD6yB3
eDsFWcn7VwADu4sB18bj/3fRs0d4r4ZoIZq/CuKkLiaYWmFFJUH2pw55iCyB66ia
iAktse5MxIoCbVQmWg3dX2kcofBq6t/hqUR3fzYfWbaZ2/T2zv+WItqlmVwTRr1O
PvdvsQKBgQDA0MQnQ7iV6Uvdm2AFHXlnwtXGelG0EsaNabF31LosHujMf92kUzaC
12NiuE0gEfuPemEw+MFQ/nfBiEDv23NgRz6frZvcvUYWr/xFs8AwPkV5pBFNwoa2
4lYPbMB/1RWf4zDKvDBKEpEGr3pCKBjrPszi0skn2DTCcpiqjH3XGg==
-----END RSA PRIVATE KEY-----
EOF
$ cluster/kubectl.sh delete -f enc-dply.yaml
deployment.apps "enc-nginx" deleted
$ cluster/kubectl.sh create -f enc-dply.yaml
deployment.apps/enc-nginx created
$ cluster/kubectl.sh get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
enc-nginx-7fb4578896-8grjd 1/1 Running 0 61s 10.88.0.102 127.0.0.1 <none> <none>
enc-nginx-7fb4578896-jjhz4 1/1 Running 0 61s 10.88.0.100 127.0.0.1 <none> <none>
enc-nginx-7fb4578896-jv9n5 1/1 Running 0 61s 10.88.0.101 127.0.0.1 <none> <none>
# test with curl
$ curl -v 10.88.0.92
< HTTP/1.1 200 OK
< Server: nginx/1.15.12
< Date: Thu, 27 Feb 2020 09:47:44 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
< Connection: keep-alive
< ETag: "5cb5d3c3-264"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
The main integration points for Encrypted Container Images
Image encryption tool
For image encryption tools, the community plans to support: docker CLI(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714), containerd imgcrypt, skopeo as well as buildah , containerd imgcrypt and skopeo have been implemented so far
mirror repository
Docker Distribution > = v2.7.1 supports image encryption OCI format
Container Runtime
For Container Runtime, the community program supports: containerd(+CRI-Containerd )as well as cri-o (Docker is not supported for now), containerd( CRI-Containerd is currently merging a PR ) and cri-o
Kubernetes
Currently, the community plans to support two image encryption modes for Kubernetes:
- Node Key Model: Put the key on the Kubernetes worker node, and use the node as the decryption granularity (CRIO has been implemented, CRI-Containerd will be merged PR)
- Multitenant Key Model: Multitenant model, which supports decryption granularity for clusters and users (not yet implemented, based on this KEP)
In addition, for Node Key Model mode, it is feasible to test the encryption process and evaluate the performance as follows:
- 1. At present, the community has no performance test data for image encryption.
- 2. Use the symmetric encryption algorithm (i.e. AES) to encrypt the image Layer, and the data length remains unchanged after encryption - it has little impact on the image pull & fast decryption speed and less time consumption
- 3,Memory would not be that bad since we are using a stream cipher
Supplement: The current community's reference to Docker's support plan is as follows (my thinking): it should wait until Docker's underlying code is switched to Containerd(moby/moby#38043), and then modify docker-shim to support image encryption, so it is currently blocked state
Conclusion
Among the contents involved in container security, the currently immature part is image encryption. Image encryption is very important in fields with strong data confidentiality requirements, such as finance, banking, and state-owned enterprises.
This paper first introduces the OCI specification involved in image encryption, and on this basis, describes the principle and process of image encryption, and uses containerd imgcrypt The tool implements the encryption process. Kubernetes supports Node Key Model and Multitenant Key Model for image encryption. This paper explains in detail the principle and process of kubernetes based on Node Key Model, and demonstrates the usage of image encryption kubernetes native in combination with CRIO
Although the current community-based work can use the image encryption feature on Kubernetes, the support for this feature is not perfect. For example, for encryption tools, it is necessary to support docker CLI(To integrate into docker CLI, we are currently waiting on moby/moby#38043. Tracking with moby/buildkit#714 ); for container runtimes, support is also required CRI-Containerd As well as Docker, etc.; for cloud native ecosystems, more cooperation with mirror-related projects (such as Notary, Clair and so on) is required; for Kubernetes, more Key Models need to be supported
Refs
- Encrypting container images with containerd imgcrypt!
- The main integration points for Encrypted Container Images
- current process
- How Encrypted Images brings about compliance in Kubernetes (via CRI-O)
- Support for Encrypted Images in Kubernetes
- Add Image Decryption KEP
- What's the difference between runc, containerd, docker? Well, I asked myself the exact same question...
- Container Runtimes Part 1: An Introduction to Container Runtimes
- Encrypted container images for container image security at rest
- Running CRI-O on kubernetes cluster
- Encrypting container images with skopeo
- OCI Container Tools Guide
- OCI Image Implementations
- OCI Runtime Implementations
- OCI Image Manifest Specification
- CRI: the Container Runtime Interface