Flux – security

Blog: Second Flux Security Audit has concluded

Thu, 09 Nov 2023 00:00:00 +0000

Precisely 2 years after performing our first security Audit, we had the chance to put Flux through a second audit this year, again facilitated by the CNCF and the Open Source Technology Improvement Fund. Trail of Bits partnered with us this time to make Flux even more secure. Flux passed the “General Availability” milestone earlier this year and the focus was on the features shipped in the Flux GA release.

The Flux maintainers and community are very grateful for the work put into this by everyone and the opportunity to grow and improve as a project. Thanks to Trail of Bits, notably Maciej Domański, Sam Alws, Sam Greenup and Jeff Braswell, who have always been extremely responsive during the process.

No new CVEs

Good news first: No new CVEs have been published for Flux in response to this second audit. Trail of Bits highlight that they found Flux was “well structured and generally written defensively” and the “audit uncovered only low- and informational-severity findings”, 10 in total. 8 of the discovered issues have been fixed as of publication of this announcement. From the remaining two issues to be fixed, one is in the process of being resolved and for the other one we have decided to accept the very low risk due to reasons mentioned in the report.

The assessment was kicked off with a list of 23 questions to be answered, circling around potential data leaks, security documentation, access control or denial of service vulnerabilities. Since the focus was on the GA components, the following parts of Flux have been put under scrutiny:

source-controller
kustomize-controller
notification-controller
Flux CLI
The pkg library, and git/gogit/fs in particular

Details on the discovered issues

You will find the full report here. The following table shows all the findings together with links to the pull requests fixing them:

Issue	Severity	Fix
1: SetExpiration does not set the expiration for the given key	low	source-controller#1185
2: Inappropriate string trimming function	informational	notification-controller#590
3: Go’s default HTTP client uses a shared value that can be modified by other components	low	flux2#4182
4: Unhandled error value	informational	flux2#4181
5: Potential implicit memory aliasing in for loops	informational	source-controller#1257, notification-controller#627, flux2#4329
6: Directories created via os.MkdirAll are not checked for permissions	informational	n/a
7: Directories and files created with overly lenient permissions	informational	pkg#663, pkg#681, source-controller#1276, kustomize-controller#1005, flux2#4380
8: No restriction on minimum SSH RSA public key bit size	informational	flux2#4177
9: Flux macOS release binary susceptible to dylib injection	low	in progress
10: Path traversal in SecureJoin implementation	undetermined	pkg#650, go-git/go-billy#31, go-git/go-billy#34

In addition to the pull requests linked above we also enabled security and quality CI checks through CodeQL via flux2#4121 to prevent any avoidable regressions.

Conclusion and next steps

From our perspective as Flux maintainers, 2 years feel like a lifetime. We added lots of new features and fixed even more bugs in that timeframe. That’s why we are particularly grateful that CNCF and OSTIF gave us the opportunity to let a team of security experts assess Flux another time. We are proud of having been able to learn from the first assessment and kept on making Flux more and more secure over these past 2 years, leading to only low- and informational-severity security findings within the GA components of Flux.

Our next milestone is the general availability of Flux’s Helm features and the subsequent general availability of the remaining Flux components. If you are interested in contributing to this, we are very much looking forward to working with you. We welcome contributions in helping resolve issues of the road, additional comments on our security posture and also welcome contributions in the form of extending our fuzzing infrastructure. Finally, if you have any additional security feedback, please come and talk to us.

Again we would like to thank the Cloud Native Computing Foundation for sponsoring the audit, the Open Source Technology Improvement Fund for the coordination and Trail of Bits for the careful review and advice during the audit period.

We are happy and proud to be part of this community!

Blog: Verify the integrity of the Helm Charts stored in OCI-compliant registries as OCI artifacts

Mon, 14 Nov 2022 10:30:00 +0000

Cosign integration was one of the most important features we shipped in the Flux v0.35 release. After that, we wrote a blog post which explains how to use the feature with OCIRepository resources which enables fetching OCI artifacts from container registries. If you haven’t read it yet, we highly encourage you to go and check it out first.

Flux v0.36.0 allows you to prove the authenticity of HelmChart resources with the help of the cosign integration. Here we will demonstrate how to use the cosign integration to verify the integrity of the Helm charts stored in OCI-compliant registries as OCI artifacts.

Not at #kubecon so I had time to prepare Flux and Flagger releases. Flagger v1.24 comes with signed releases & OCI Helm charts. Flux v0.36 adds support for verifying Helm charts with Cosign. https://t.co/6MYwzfMA3W
— Stefan Prodan (@stefanprodan) October 27, 2022

Starting with Helm v3.8.0, Helm supports the OCI registry as a one of the storage option for Helm charts as an alternative to Helm repositories. The Helm CLI can push and pull Helm charts to and from OCI-compliant registries.

Note: Prior to Helm v3.8.0, OCI support was experimental. To use it there, you need to enable the feature by setting the HELM_EXPERIMENTAL_OCI environment variable to 1.

As we store Helm charts in OCI-compliant registries as OCI artifacts, we can now use the cosign integration to sign and verify them. Also, thanks to Flux, you can reconcile resources such as plain-text Kubernetes YAML manifests, Terraform modules, etc. from OCI-compliant registries with the help of OCIRepository resources. You can achieve the same thing for Helm charts with HelmRepository resources. This means that you can store Helm charts in OCI-compliant registries as OCI artifacts and use Flux to reconcile them like the following:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
 name: podinfo
 namespace: default
spec:
 type: "oci"
 interval: 5m0s
 url: oci://ghcr.io/stefanprodan/charts

Note:

Here you can review the complete registries lists that support the OCI artifact specification: OCI-Conformant Products.

You will notice that when you open the list, DockerHub is not included into the list but it will be added soon because they recently announced OCI Artifacts support, and you can read more about it from here.

Let’s jump right into the details of how we can actually use it.

We will deploy Prometheus by using its community Helm charts stored as OCI artifacts in OCI registry. Recently, Prometheus’ community started to publish their Helm charts to OCI registries and sign them with cosign using the keyless approach, you can learn more the process here. Then we are going to verify it with cosign and configure Flux to verify the Helm chart’s signatures before they are downloaded and reconciled. As the Prometheus community signed their Helm Charts without providing a key pair, we do not need to specify any key in the HelmChart resource’ provider.cosign spec to enable keyless verification for Flux.

For the sake of simplicity, we’ve deployed Prometheus alone but if you want to learn more about installing the Prometheus stack including Grafana, Alertmanager, etc., please refer to the official Flux page that can help you to do that.

You need three things to complete this demo;

cosign CLI
- https://docs.sigstore.dev/cosign/installation/
A Kubernetes cluster
- https://kind.sigs.k8s.io/#installation-and-usage
Flux CLI
- https://fluxcd.io/flux/cmd/

Let’s start by creating a simple Kubernetes cluster:

kind create cluster

Use the Flux CLI to do pre-flight checks:

$ flux check --pre
► checking prerequisites
✔ Kubernetes 1.25.3 >=1.20.6-0
✔ prerequisites checks passed

If the checks are successful, you can install Flux on the cluster.

Let’s install Flux on it - if you need to use other options, check out the installation page.

export GITHUB_USER=developer-guy
export GITHUB_TOKEN=$GITHUB_TOKEN

flux bootstrap github \
 --owner=$GITHUB_USER \
 --repository=flux-cosign-helm-oci-demo \
 --branch=main \
 --path=./clusters/my-cluster \
 --personal

Note: Don’t forget to change the values with your own details!

As we stick to GitOps practices, we only create files that contain the HelmRepository and HelmRelease resources. After committing and pushing those changes into the upstream repository, Flux will watch for changes and use them as source-of-truth for the configuration:

git clone git@github.com:developer-guy/flux-cosign-helm-oci-demo.git
cd flux-cosign-helm-oci-demo

Let’s create the HelmRepository resource first:

flux create source helm prometheus-community \
 --url=oci://ghcr.io/prometheus-community/charts \
 --interval=10m \
 --export > ./clusters/my-cluster/prometheus-community-helmrepository.yaml

Now, let’s move on with creating the HelmRelease resource:

flux create helmrelease prometheus \
 --source=HelmRepository/prometheus-community \
 --chart=prometheus \
 --interval=10m \
 --release-name prometheus \
 --target-namespace=monitoring \
 --create-target-namespace \
 --export > ./clusters/my-cluster/prometheus-helmrelease.yaml

and run the following command to add the verify section to the HelmRelease resource’ .spec.chart.spec section to enable keylesss verification:

yq e '.spec.chart.spec|=({"verify": { "provider": "cosign" } } +.)' ./clusters/my-cluster/prometheus-helmrelease.yaml

This command above will add the following part to the HelmRelease resource’s .spec.chart.spec section:

verify:
 provider: cosign

then, commit and push the changes:

git commit -m "Add prometheus HelmRelease and HelmRepository resources"
git push

After a couple of seconds, Flux will have applied these changes. Now let’s check the status of them:

Or, you can trigger the reconciliation immediately by running the simple command: flux reconcile source git flux-system

For the HelmRepository resource:

$ flux get sources helm
NAME REVISION SUSPENDED READY MESSAGE
prometheus-community False True Helm repository is ready

For the HelmRelease resource:

$ flux get helmreleases
NAME REVISION SUSPENDED READY MESSAGE
prometheus 15.18.0 False True Release reconciliation succeeded

If everything is fine, you can check the pods in the monitoring namespace:

$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
prometheus-alertmanager-54b7d7cf45-2b7zf 2/2 Running 0 115s
prometheus-kube-state-metrics-67f68d64bb-vlmvd 1/1 Running 0 115s
prometheus-node-exporter-46gm6 1/1 Running 0 115s
prometheus-pushgateway-596cd99697-t79zt 1/1 Running 0 115s
prometheus-server-c458cf6f9-nvstw 2/2 Running 0 115s

Great! Now, you have installed Prometheus with Flux by using the Helm chart stored in the OCI registry and verified it with cosign.

We can assume that the Helm chart’s signature is verified as we let it be deployed in a cluster but let’s do have double check and check the status of the HelmRelease to see whether the verification is successful or not:

$ kubectl get helmcharts -n flux-system flux-system-prometheus -ojsonpath='{.status.conditions[?(@.type=="SourceVerified")]}'
{"lastTransitionTime":"2022-11-09T13:27:38Z","message":"verified signature of version 15.18.0","observedGeneration":1,"reason":"Succeeded","status":"True","type":"SourceVerified"}

That’s super cool! Because Flux is going to add a condition to the HelmChart resource’s status section to show the verification status. If the verification is successful, it will add a condition like the one above.

DIY (Do it yourself) Approach

The Prometheus community Helm charts only serve as an example. Here is how you can do the same thing with your own Helm charts.

Create a Helm chart
Package the Helm chart as .tar.gz file
Login to the OCI-compliant registry that you want to use to store your Helm chart
Push the Helm chart as OCI artifact

Let’s create a sample directory that will contain our Helm chart:

mkdir -p helm-oci-demo
cd helm-oci demo

Now package the Helm chart as .tar.gz file:

 helm create nginx

Let’s package the Helm chart as .tar.gz file:

helm package nginx

Now, let’s login to the OCI-compliant registry that you want to use to store your Helm chart. In this example, we’ll be using the GitHub Container Registry (ghcr.io):

echo $GHCR_PAT | helm registry login ghcr.io -u $USER --password-stdin

Note: Don’t forget to change the values with your own details!

At this point, we are ready to push the Helm chart as OCI artifact:

$ helm push nginx-0.1.0.tgz oci://ghcr.io/$USER
Pushed: ghcr.io/developer-guy/nginx:0.1.0
Digest: sha256:21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

So, the Helm chart is pushed to the OCI registry. It’s time to sign it with cosign. As cosign recommends we should always sign images based on their digests (@sha256:) rather than a tag. So, we should grab the digest from the command output above which is 21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7 in this case, and use that digest while signing the image:

As we saw the keyless approach before, let’s try the key-based approach this time. To do that, we should create public/private key pairs first:

cosign generate-key-pair

This command will generate two files, a cosign.pub which is a publickey and cosign.key which is a private key pair and store them in the current directory directory.

Now, let’s sign the image with the private key:

cosign sign --key cosign.key ghcr.io/$USER/nginx@21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

Cool! Now we have signed the image with the private key. Let’s check the signature:

cosign verify --key cosign.key ghcr.io/$USER/nginx@21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

Yay! It’s verified. But in order to make the public key accessible by Flux, we need to create a Kubernetes secret to store the public key:

kubectl -n flux-system create secret generic cosign-pub \
 --from-file=cosign.pub=cosign.pub

Now, we can use it in our Flux configuration. The rest of the steps are the same as the previous section. For the sake of simplicity, we won’t repeat them here other than the HelmRepository and HelmRelease resources’ creation steps.

Let’s create the HelmRepository resource first:

flux create source helm $USER-charts\
 --url=oci://ghcr.io/$USER \
 --interval=10m \
 --export > ./clusters/my-cluster/nginx-helmrepository.yaml

Let’s move on with creating the HelmRelease resource:

flux create helmrelease nginx \
 --source=HelmRepository/$USER-charts \
 --chart=nginx \
 --interval=10m \
 --release-name nginx \
 --target-namespace=default \
 --export > ./clusters/my-cluster/nginx-helmrelease.yaml

Don’t forget to run the following command to add the verify section to the HelmRelease resource’ .spec.chart.spec section to enable verification:

yq e '.spec.chart.spec|=({"verify": { "provider": "cosign", "secretRef": { "name": "cosign-pub" } } } +.)' ./clusters/my-cluster/nginx-helmrelease.yaml

This command above will add the following part to the HelmRelease resource’s .spec.chart.spec section:

verify:
 provider: cosign
 secretRef:
 name: cosign-pub

That’s all you need to do folks!

Congratulations! You have successfully signed your Helm chart with cosign with key-based approach and used it with Flux.

Blog: Prove the Authenticity of OCI Artifacts

Mon, 17 Oct 2022 12:30:00 +0000

Software supply chain attacks are one of the most critical risks threatening today’s software and have begun to collapse like a dark cloud over the software industry. For the Flux family of projects we are taking precautions against these threats. Apart from implementing security features and best practices, it is important to us to educate our users. You can find all Flux’s security articles here. Today we will talk about a new security feature.

Let’s start with a brief historical explanation of how we got to this point. It all started with the following sentence:

Flux should be able to distribute and reconcile Kubernetes configuration packaged as OCI artifacts.

RFC-0003: Flux OCI support for Kubernetes manifests.

From then on, the Flux community worked hard and brought this feature with Flux v0.32. So with that, you can store and distribute various sources such as Kubernetes manifests, Kustomize overlays, and Terraform modules as OCI (Open Container Initiative) artifacts with Flux CLI and tell Flux to reconcile your sources that are stored in OCI Artifacts, and Flux will do that for you. 🕺🏻

But this only covered the first stage of the entire implementation. There is more than that. ☝️

One of the most exciting features of this RFC is the verification of artifacts. But why, what is it, is it really necessary or just a hype thing? This is a long topic that we need to discuss. Suppose you store the cluster desired state as OCI artifacts in a container registry. How can you be one hundred percent sure that the resources that Flux reconciles are the same as the resources that you’ve pushed to the OCI registry? This is where the verification of artifacts comes into play. But, how can we do that? 🤔

Thanks to the Sigstore community we have a great set of services and tools for signing and verifying authenticity. One of the tools is cosign which can be used for container signing, verification, and storage in an OCI registry. We will use it to verify the authenticity of the OCI Artifacts in Flux. Starting with v0.35, Flux comes with support for verifying OCI artifacts signed with Sigstore Cosign. Documentation for setting it up can be found here.

Let’s jump right into the details of how we can actually use it.

We will deploy cert-manager by storing its manifests in OCI registry packaged as an OCI Artifacts, using the Flux CLI. Then we are going to sign it with cosign and configure Flux to verify the artifacts’ signatures before they are downloaded and reconciled.

You need three things to complete this demo;

cosign CLI
- https://docs.sigstore.dev/cosign/installation/
A Kubernetes cluster
- https://kind.sigs.k8s.io/#installation-and-usage
Flux CLI
- https://fluxcd.io/flux/cmd/

Let’s start by creating a simple Kubernetes cluster:

kind create cluster

Let’s install Flux on it - if you need to use other options, check out the installation page.

export GITHUB_USER=developer-guy
export GITHUB_TOKEN=$GITHUB_TOKEN

flux bootstrap github \
 --owner=$GITHUB_USER \
 --repository=flux-cosign-demo \
 --branch=main \
 --path=./clusters/my-cluster \
 --personal

⚠️ Note: Don’t forget to change the values with your own details!

First we download the cert-manager install manifests from GitHub:

curl -sSLO https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

https://github.com/cert-manager/cert-manager/releases/tag/v1.9.1

Next we push the manifests to GitHub container registry with Flux CLI:

mkdir -p ./manifests
 cp cert-manager.yaml ./manifests
$ flux push artifact oci://ghcr.io/$GITHUB_USER/manifests/cert-manager:v1.9.1 \
 --path="./manifests" \
 --source="https://github.com/cert-manager/cert-manager.git" \
 --revision="v1.9.1/4486c01f726f17d2790a8a563ae6bc6e98465505"
► pushing artifact to ghcr.io/developer-guy/manifests/cert-manager:v1.9.1
✔ artifact successfully pushed to ghcr.io/developer-guy/manifests/cert-manager@sha256:d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b

Before signing the OCI artifact with Cosign, we need to create a set of key pairs, a public and private one:

cosign generate-key-pair

This command above outputs two files to disk: cosign.pub and cosign.key. The cosign.pub file is the public key and the cosign.key file is the private key. You can use the cosign.pub file to verify the container image and the cosign.key file to sign the container image.

To let Flux to verify the signature of the OCI artifact, we should create a secret that contains the public key::

kubectl -n flux-system create secret generic cosign-pub \
 --from-file=cosign.pub=cosign.pub

Now, let’s sign it:

$ cosign sign --key cosign.key ghcr.io/$GITHUB_USER/manifests/cert-manager:v1.9.1
Enter password for private key:
Pushing signature to: ghcr.io/developer-guy/manifests/cert-manager

As we stick into the GitOps practices, we should create a file that contains the OCIRepository resource, then commit and push those changes into the upstream repository that Flux watches for changes:

git clone git@github.com:developer-guy/flux-cosign-demo.git
cd flux-cosign-demo

Let’s create a secret with the GitHub token:

$ flux create secret oci ghcr-auth \
 --url=ghcr.io \
 --username=${GITHUB_USER} \
 --password=${GITHUB_TOKEN}
► oci secret 'ghcr-auth' created in 'flux-system' namespace

Configure Flux to pull the cert-manager artifact, verify its signature and apply its contents:

cat << EOF | tee ./clusters/my-cluster/cert-manager-sync.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
 name: cert-manager
 namespace: flux-system
spec:
 interval: 5m
 url: oci://ghcr.io/${GITHUB_USER}/manifests/cert-manager
 ref:
 semver: "*"
 secretRef:
 name: ghcr-auth
 verify:
 provider: cosign
 secretRef:
 name: cosign-pub
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
 name: cert-manager
 namespace: flux-system
spec:
 interval: 1h
 timeout: 5m
 sourceRef:
 kind: OCIRepository
 name: cert-manager
 path: "."
 prune: true
 wait: true
EOF

Let’s commit and push these changes:

git add .
git commit -m"Add cert-manager OCIRepository and Kustomization"
git push

After couple of seconds for Flux will have applied these changes. Now let’s check the status of them:

Or, you can trigger the reconcilation immediately by running the simple command: flux reconcile source git flux-system

For Kustomization resources:

$ flux get kustomizations
NAME REVISION SUSPENDED READY MESSAGE
cert-manager v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b False True Applied revision: v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b
flux-system main/14f1e66 False True Applied revision: main/14f1e66

For OCIRepository resources:

$ flux get sources oci
NAME REVISION SUSPENDED READY MESSAGE
cert-manager v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b False True stored artifact for digest 'v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b'

If you see the status of the OCIRepository is True, it means that Flux has successfully verified the signature of the container image. Because Flux adds a condition with the following attributes to the OCIRepository’s .status.conditions:

type: SourceVerified
status: “True”
reason: Succeeded

If the verification fails, Flux will set the SourceVerified status to False and will not fetch the artifact contents from the registry. If you see the status of the Kustomization is True, it means that Flux has successfully applied the manifests that are stored in the container image.

Let’s check the status of the cert-manager deployment:

$ kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-cainjector-857ff8f7cb-l469h 1/1 Running 0 76s
cert-manager-d58554549-9fbgj 1/1 Running 0 76s
cert-manager-webhook-76fdf7c485-9v82g 1/1 Running 0 76s

Furthermore

As we can store Helm Charts in OCI registries with the release of Helm v3.8.0 which means that we can also sign them with cosign and verify them with Flux. The Flux community is already working on it and want to add support for verifying the Helm charts stored in OCI registries as OCI Artifacts in the next releases of Flux. You can follow the progress of this feature in the following issue: fluxcd/source-controller#914.

The Sigstore community is aware of the risks and the toil of managing public/private key pairs, so cosign offers another mode for signing and verification called Keyless, which do not require managing any keys manually. Flux also supports that. If you omit the .verify.secretRef field, Flux will try to verify the signature using the Keyless mode. It’s worth mentioning keyless verification is an experimental feature, using custom root CAs or self-hosted Rekor instances are currently not supported.

Blog: May 2022 Security Announcement

Tue, 10 May 2022 08:30:00 +0000

tl;dr

The Flux Team has found three security vulnerabilities in Flux, and we strongly advise you to upgrade your clusters as soon as you can.

CVE	Advisory	Severity	Affected versions
CVE-2022-24817	Improper kubeconfig validation allows arbitrary code execution	Critical	`< 0.29.0 >= v0.1.0`
CVE-2022-24877	Improper path handling in Kustomization files allows path traversal	Critical	`< v0.29.0`
CVE-2022-24878	Improper path handling in Kustomization files allows for denial of service	High	`< v0.29.0 >= v0.19.0`

Breaking changes to be aware of in the upgrade process: 0.29, 0.28, 0.27, 0.26, 0.24 - 0.21.

If you cannot immediately update or are hard pressed for time and need a work-around for now, please see the CVE advisories linked above for more information.

Some Background

Last week the Flux Security Team disclosed three new vulnerabilities which affect v0.28 and older versions, and have a greater impact in multi-tenancy deployments.

The reason why the impact is greater in multi-tenancy deployments is due to the way that Flux/GitOps works. Flux tends to operate like a cluster admin, having permissions to apply any changes to a cluster, regardless of their scope - at namespace or cluster levels. Users having access to a source repository, or simply having access to create/alter Flux objects within a cluster can instruct Flux to apply such changes, which in single tenancy effectively means that such users have cluster admin permissions to the target clusters. The caveat being that using Flux you can add additional security controls between the user and the target cluster, for example, before each change is merged into a repository Pull Requests must be created requiring peer reviews. The Open GitOps community started defining and codifying this further - have a look at this blog post if you want to know more.

In multi-tenant environments, users with similar permissions can only affect part of a cluster, or an isolated cluster in a group of clusters. Therefore, if a user can gain escalated privileges (or deny service) at a Flux level, it will have an impact way larger than on single-tenant clusters, as the users can impact more than just themselves.

At time of writing, we are unaware of any public exploits in the wild, and therefore have no reason to believe such vulnerabilities have been actively exploited.

The advisories in detail

CVE-2022-24817 - Kubeconfig Validation

At the beginning of the Flux2’s journey we implemented a feature to apply state to remote clusters. It enables users to have a management cluster in which Flux is installed, which then applies changes to other clusters, making it ideal for some multi-tenancy scenarios in which high isolation across tenants is needed.

The connection between the management cluster and the target clusters is done by referencing a Kubernetes secret containing a kubeconfig, which is set at the spec.KubeConfig field in either Kustomization or HelmRelease objects.

One interesting thing about kubeconfigs is that they are quite extensible. You can for example define an executable which will then be automatically called by kubectl every time it requires a new on-demand access token. An example of this in action is aws-iam-authenticator, which enables AWS users to authenticate against AWS and then use the returning JWT tokens to access their EKS clusters. Once the token expires, the process happens again. All that is managed by kubectl behind the scenes without user intervention.

The problem here is that first, the use of executables in kubeconfigs was enabled by default. Meaning that a malicious tenant would be able to craft a malicious kubeconfig which could lead to privilege escalation within the cluster.

As a solution, we decided to disable this feature by default. It can still be enabled at a cluster level via a new flag --insecure-kubeconfig-exec being sent to the controller binary.

For cluster admins considering this feature, we also recommend the use of AppArmor and SELinux profiles to enforce at Kernel level what binaries could be executed.

CVE-2022-24877 - Kustomization Path Traversal

Flux allows users to lean on Kustomize features to make their lives easier as they go on about declaring the state of their clusters. Some of those features could result in sensitive data from the pod filesystem to be exposed into the target cluster, which could lead to a malicious tenant being privy to anything sensitive that may exist in the controller’s filesystem or attached volumes (e.g. token).

The mitigation for the path traversal was to create stronger bounds, enforcing that all kustomize operations happen within such bounds or result in error.

CVE-2022-24878 - Kustomization Denial of Service

Whilst working on the mitigation of the previous CVE, we have noticed that in some scenarios a specially crafted kustomization.yaml could lead to the kustomize-controller to enter into an endless loop, and finally crash.

For single-tenant clusters, this would mean that an user may make a mistake and the controller stops working, potentially resulting in future reconciliations not being applied. For multi-tenant clusters, depending on the deployment model, a tenant could cause a disruption to affect not only itself but also the other tenants and potentially even the management cluster.

The solution mitigating this vulnerability was to further improve our validation and ensure that such scenarios are not processed in the first place.

Inspecting your Flux system version

To check if your system is currently vulnerable, run flux version --context=my-cluster with the --context set to the cluster you want to inspect. This will report the current Flux binary and controller versions.

Vulnerable Flux system

To find out if your system could be vulnerable, simply find out the version of Flux. Here it’s important to check you are running all of these:

flux < v0.29.0
helm-controller < v0.19.0
kustomize-controller < v0.24.0.

You can do this like so:

$ flux version
flux: v0.22.1
helm-controller: v0.12.2
kustomize-controller: v0.17.0
notification-controller: v0.18.1
source-controller: v0.17.2

Updating your vulnerable system

💥 If you find the controllers versioned within the range mentioned above, follow the upgrade procedure for your system. For flux bootstrap, this can be done by running the command again with the same arguments as used during install.

⚠️ Please note that if you are upgrading from below one of the versions in the following list, there are breaking changes and, pre- and/or post-upgrade notes you need to take into account: 0.29, 0.28, 0.27, 0.26, 0.24 - 0.21.

Up-to-date Flux system

An up to date Flux system should at least have versions listed below:

flux >= v0.29.0
helm-controller >= v0.19.0
kustomize-controller >= v0.24.0

So in practice the output could look like this:

$ flux version
flux version
flux: v0.30.2
helm-controller: v0.21.0
kustomize-controller: v0.25.0
notification-controller: v0.23.5
source-controller: v0.24.4

We encourage all users to keep Flux up-to-date. We offer a GitHub Action with which you can automate the Flux upgrades in a GitOps manner, without having to connect from CI to the cluster’s API, as Flux is capable of upgrading itself from Git.

Flux Security more generally speaking

It is no secret that the Flux community has been investing in security for a long time. Early on the Flux journey we re-architectured the codebase to avoid shelling out to binaries to decrease the likelihood of code execution vulnerabilities. The story about our Git integration we wrote down here.

As Security is such a central pillar of Flux, we are keen to write about it and tell you how you can benefit from all the individual features and improvements we worked on, e.g. SBOMs, CI Checks, Branch Protection, restricted pod security standard and more. Since the beginning we worked hard to ensure that we ship code that does what it needs to do, even when that means having to rewrite parts of upstream dependencies.

When we had our first security audit last year, the results were quite reassuring as most of the findings were quite small, with the exception of a RCE in kustomize-controller, which speaks to the security improvements we have been investing on, and how they are resulting in better development practices.

Another recommendation from the auditors was to implement and follow a stricter and more elaborate RFC process, which is what we did. They also recommended we get in touch with other security teams or auditors for getting feedback on a refined and more general multi-tenancy proposal - which we also did as mentioned below.

What’s next for Flux

The way we shaped the Flux Roadmap for GA was

Feature party with Flux Legacy
Stable APIs (this was after the controller refactoring which consolidated functionality in fluxcd/pkg)
Straightforward multi-tenancy implementation
GA release

Here is the status quo regarding multi-tenancy:

Flux2 supports multi-tenancy, and users have been using it in production for some time now.

The documentation around the subject covers a bootstrap example to help users kick start their multi-tenancy deployments. And also how to implement control plane isolation with the multi-tenancy-lockdown.

What's next

In summary, the documentation needs expanding to better inform users around the security risks of multi-tenancy and the recommended deployment models for their specific isolation/security requirements.

There are proposed changes that would further improve Flux in multi-tenancy environments, by for example enabling tenants to share resources amongst themselves. Such changes must be progressed once the security impact of such changes have been assessed.

To help us get this right, we are engaging with the CNCF TAG Security. This is the upstream group where key contributors and experts of the CNCF Landscape assemble and define security best practices across all the individual Cloud Native projects. We are asking them for an independent security review and recommendations, particularly around multi-tenancy.

If you want to join the conversation, we are all ears. Please refer to the open RFC documents and have your say there. We definitely want to get this right for everyone.

In addition to that we are working hard to round up features, improve their performance, security and overall stability.

If you want to follow all the other GA related work, we explained how to do that here and if you would like to participate in any of the discussions, come and find us on Slack or in our regular meetings. We are always looking forward to growing Team Flux and the closer we get to GA, it’s getting even more important to have all voices heard.

Blog: Security: Using Pod Security Standard "restricted"

Wed, 09 Mar 2022 12:30:00 +0000

Next up in our blog series about Flux Security is how we moved to Pod Security Standard “restricted”, all the background info you need to know and how that makes things safer for you.

Since version 0.26 of Flux we are applying

[..] the restricted pod security standard to all controllers. In practice this means:

all Linux capabilities were dropped

the root filesystem was set to read-only

the seccomp profile was set to the runtime default

run as non-root was enabled

the filesystem group was set to 1337

the user and group ID was set to 65534

Flux also enables the Seccomp runtime default across all controllers. Why is this important? Well, the default seccomp profile blocks key system calls that can be used maliciously, for example to break out of the container isolation. The recently disclosed kernel vulnerability CVE-2022-0185 is a good example of that.

Pod Security Standards definition

Kubernetes defined three policies in its Pod Security Standards. They range from

Privileged: This does not place any restrictions on the workload at all. The idea being that this can be used for system- and infrastructure-level workloads which are managed by privileged and trusted users only.
Baseline: This policy comes with some restrictions. It aims to guard against known privilege escalations while still making it easy to adopt and use it by keeping a certain level of compatibility with most workloads.
Restricted: Inheriting all the restrictions from Baseline, it enforces additional limitations, thus follows hardening best practices by increasing the isolation levels the workload is exposed to.

We are very pleased that all Flux controllers were moved to Restricted, as that offers the highest level of security for you.

We recommend checking out the Upstream Kubernetes documentation on Pod Security Standards as it gives a generally good overview of all the security features enabled. In addition to that you can see which restrictions were added as part of which Kubernetes release, meaning that with every Kubernetes release, you will benefit from new Upstream Kubernetes security improvements automatically.

Note:

As of v1.24 Kubernetes still runs all workloads with seccomp in unconfined mode, in other words, disabled. On the other hand, Docker has seccomp enabled by default for years now.

There are discussions to change the Kubernetes default on v1.25, and have all workloads set to RuntimeDefault unless opted-out. This would be based on SeccompDefault feature gate being enabled from that version onwards.

Note: If you are an OpenShift user, you might run into this issue ( related upstream report). The work-around right now is to remove the seccomp profile as described in these instructions.

`seccomp` and `RuntimeDefault`

Seccomp is short for “Secure Computing”. It refers to a facility in the Linux kernel which can limit the number of system calls available to a given process. Right now there are around 300+ system calls available, e.g. read to read from a file descriptor or chmod to change the permissions of a file. The more syscalls you block, the more secure your application, as a rogue process will only be able to do what you specified.

In its first inception seccomp was introduced into Linux in 2005, to Docker in version 1.10 (Feb 2016) and to Kubernetes in version 1.3 (Jul 2016). So while the technology has been around for a while and you could handcraft your own seccomp profiles, the challenge has always been striking the right balance: if you are too generous in your filter, it won’t guard against malware effectively – if you are too strict, your application might not work.

All container runtimes come with a default seccomp profile. Docker Desktop for example blocks around 44 system calls. In Kubernetes you can enable the seccomp profile RuntimeDefault for your pod like so:

spec:
 securityContext:
 seccompProfile:
 type: RuntimeDefault

All Flux controllers have this implemented as well now!

By adopting both changes, we further restrict the permissions that Flux requires in order to operate. This, alongside other changes we are working on, translate in a decreased attack surface which may reduce the impact of eventual CVEs that may surface in our code base - or our supply chain.

Talk to us

We love feedback, questions and ideas, so please let us know your personal use-cases today. Ask us if you have any questions and please

join our upcoming dev meetings
find us in the #flux channel on CNCF Slack
add yourself as an adopter if you haven’t already

See you around!

Blog: Security: More confidence through Fuzzing

Tue, 22 Feb 2022 08:30:00 +0000

Next up in our blog series about Flux Security is how we implemented fuzzing in Flux and its controllers and how that makes things safer for you.

Wikipedia explains Fuzzing like so:

Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks. Typically, fuzzers are used to test programs that take structured inputs. This structure is specified, e.g., in a file format or protocol and distinguishes valid from invalid input. An effective fuzzer generates semi-valid inputs that are "valid enough" in that they are not directly rejected by the parser, but do create unexpected behaviors deeper in the program and are "invalid enough" to expose corner cases that have not been properly dealt with.

We already have quite a good coverage of unit and end-to-end tests across the controllers. Adding fuzzing to the mix will further extend the scope of tests to scenarios and payloads not previously covered. Together with the fuzzing that’s already being done within the Kubernetes repositories, e.g. kubernetes, client-go and apimachinery we feel ever more confident in our code.

We are happy to share that since the 0.27 release of Flux all Flux controllers and libraries are now tested by Google’s continuous fuzzing for open source software.

How we got here

When we announced the results of the security audit back in November, we already shared that the team at ADA Logics had helped put together an initial implementation of Fuzzing for some of the Flux controllers. In this first inception three issues were already found (1x slice out-of-bounds, 2x nil-dereference), and immediately fixed. Naturally we were very interested in merging the fuzzing integration.

In order for us to fully land the fuzzers, we needed to make some architectural changes to the build process, especially for the controllers that rely on C bindings to libgit2, such as source-controller and image-automation-controller, which are now statically built. In addition to that, we extended the scope of the fuzzers considerably. If you take a look at the related pull request for notification-controller you get a good idea of what this all entailed, e.g. fuzzing for all notifiers.

Fuzzers are now run for every commit which lands in the Flux controllers and libraries.

Thanks again ADA Logics for contributing and to everyone else who helped integrate this! We are also very grateful to Google and OpenSSF who provide and maintain the required infrastructure.

What’s next

As Go will see built-in Fuzz support in 1.18, we were very interested in structuring everything closely to the new format, so that the transition from dvyukov/go-fuzz (which is currently being used) goes smoothly. (We can recommend Jay Conrod’s blog post about the Internals of Go’s new fuzzing system, if you are curious!)

The move of Flux to go native fuzzing is being tracked in this issue. We also hope to add new fuzzers soon, so if you want to contribute there: come and find us on Slack! It’s an easy way to get to know and extend the Flux codebase.

This is just one more measure we are taking to keep you more secure.

Talk to us

We love feedback, questions and ideas, so please let us know your personal use-cases today. Ask us if you have any questions and please

join our upcoming dev meetings
find us in the #flux channel on CNCF Slack
add yourself as an adopter if you haven’t already

See you around!

Blog: Security: Image Provenance

Mon, 14 Feb 2022 10:30:00 +0000

Next up in our blog series about Flux Security is how and why we use signatures for the Flux CLI and all its controller images and what you can do to verify image provenance in your workflow.

Since Flux 0.26 our Security Docs had this addition:

The Flux CLI and the controllers' images are signed using Sigstore Cosign and GitHub OIDC. The container images along with their signatures are published on GitHub Container Registry and Docker Hub.

To verify the authenticity of Flux’s container images, install cosign and run cosign verify.

We are very pleased to bring this to you and encourage you to make use of it in your workflows to make your clusters more secure. But let’s unpack everything the above section says.

Why sign release artifacts in the first place?

Essentially we want you to be able to verify Flux’s image provenance, which boils down to making sure that

The release you just downloaded actually comes from us, the Flux team
It hasn’t been tampered with

Cryptographic signatures are the go-to choice for this and have been used for decades, but are not without challenges.

The (excellent) sigstore docs say this:

Software supply chains are exposed to multiple risks. Users are susceptible to various targeted attacks, along with account and cryptographic key compromise. Keys in particular are a challenge for software maintainers to manage. Projects often have to maintain a list of current keys in use, and manage the keys of individuals who no longer contribute to a project. Projects all too often store public keys and digests on git repo readme files or websites, two forms of storage susceptible to tampering and less than ideal means of securely communicating trust.

The tool sets we’ve historically relied on were not built for the present circumstance of remote teams either. This can be seen by the need to create a web of trust, with teams having to meet in person and sign each others’ keys. The current tooling (outside of controlled environments) all too often feel inappropriate to even technical users.

We are very happy that sigstore exists. It is a Linux Foundation project backed by Google, Red Hat and Purdue University striving to establish a new standard for signing, verifying and provenance checks for the open source community.

The way it works is that our cosign workflow uses

cosign to sign our release artifacts and store the signatures in an OCI registry (GHCR and Docker Hub in our case)
OpenID Connect (OIDC) beforehand to be identified via our email address
Fulcio, a root certification authority (CA), which issues a time-stamped certificate for the identified user that has been authenticated
Rekor, acts as the transparency log storage, where the certificate and signed metadata is stored in a searchable ledger, that can’t be tampered with

That’s a lot of terminology and project names, but what’s beautiful about cosign is that you can relatively easily integrate this using GitHub Actions (see how it was done in source-controller).

How to verify signatures

If you want to do this manually as an one-off, install the cosign tool and essentially just run cosign verify for the image you want to verify:

cosign verify ghcr.io/fluxcd/source-controller:v1.0.0-rc.5 \
 --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
 --certificate-identity-regexp=^https://github.com/fluxcd/.*$

With --certificate-oidc-issuer we verify that the Flux image was signed by GitHub Actions OIDC issuer. With --certificate-identity-regexp we verify that the image originates from the FluxCD GitHub organization.

The output should be:

Verification for ghcr.io/fluxcd/source-controller:v1.0.0-rc.5 --
The following checks were performed on each of these signatures:
 - The cosign claims were validated
 - Existence of the claims in the transparency log was verified offline
 - The code-signing certificate was verified using trusted certificate authority certificates

Now let’s take a look at how to automate this further.

Enforcing verified signatures in a cluster

Luckily cosign is compatible with and supported by policy engines such as Connaisseur, Kyverno and OPA Gatekeeper. Let’s go with Kyverno for now. To make sure that Flux image signatures are verified, all you need to do is add the following manifest:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: verify-flux-images
spec:
 validationFailureAction: enforce
 background: false
 webhookTimeoutSeconds: 30
 failurePolicy: Fail
 rules:
 - name: verify-cosign-signature
 match:
 any:
 - resources:
 kinds:
 - Pod
 verifyImages:
 - imageReferences:
 - "ghcr.io/fluxcd/source-controller:*"
 - "ghcr.io/fluxcd/kustomize-controller:*"
 - "ghcr.io/fluxcd/helm-controller:*"
 - "ghcr.io/fluxcd/notification-controller:*"
 - "ghcr.io/fluxcd/image-reflector-controller:*"
 - "ghcr.io/fluxcd/image-automation-controller:*"
 attestors:
 - entries:
 - keyless:
 subject: "https://github.com/fluxcd/*"
 issuer: "https://token.actions.githubusercontent.com"
 rekor:
 url: https://rekor.sigstore.dev

Have a look at fluxcd/flux2-multi-tenancy to see a more elaborate example and how the Kyverno policies are wired up there.

By verifying all our artifacts, you ensure their provenance and guarantee that they haven’t been modified from the moment we signed and shipped them. This is just one more measure we are taking to keep you more secure.

Talk to us

We love feedback, questions and ideas, so please let us know your personal use-cases today. Ask us if you have any questions and please

join our upcoming dev meetings
find us in the #flux channel on CNCF Slack
add yourself as an adopter if you haven’t already

See you around!

Blog: Security: The Value of SBOMs

Mon, 07 Feb 2022 08:30:00 +0000

Flux - built with security in mind

You don’t get to re-architect a successful project very often, but we did about two years ago. The Flux project was already off to a great start and had many happy adopters and many of its design principles we kept at the forefront of our mind:

Pull vs Push: if you haven’t read this great blog post from 2018 about why you want Pull - all it says still holds true.
Least amount of privileges.
Reusing best practises, libraries and tools, e.g. client-go and helm APIs.
And more we will explain further down the line.

Why did we re-architect and rewrite Flux? Flux Legacy (v1) had been started Mid-2016 and while it worked great and still does, it didn’t quite benefit from more recent developments in the Kubernetes space like controller-runtime because it pre-dated them significantly.

Also rewriting Flux as a set of very targeted controllers was a unique opportunity to reduce the scope (and thus attack surface) of these individual sub-projects and make testing and debugging a lot easier. Re-usability as well.

All of this said, we believe that a blog series about Flux and its security considerations and features is in order and we will kick it off talking about SBOMs.

What is a SBOM?

Since Flux release 0.26 we publish a SBOM for each of the individual controllers. We reported about this in the accompanying monthly update blog post.

So what is a SBOM? It’s short for Software Bill of Materials. Wikipedia defines it as

A software bill of materials (SBOM) is a list of components in a piece of software. Software vendors often create products by assembling open source and commercial software components. The SBOM describes the components in a product. It is analogous to a list of ingredients on food packaging: where you might consult a label to avoid foods that may cause allergies, SBOMs can help organizations or persons avoid consumption of software that could harm them.

The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.

For the Flux project we publish a Software Bill of Materials (SBOM) with each release. The SBOM is generated with Syft in the SPDX format.

The spdx.json file is available for download on the GitHub release page e.g.:

curl -sL https://github.com/fluxcd/flux2/releases/download/v0.25.3/flux_0.25.3_sbom.spdx.json | jq

Inspecting the JSON data, you will see that for each of the files and libraries required for building and shipping the release you can verify the license, origin, version and checksum.

What might seem like a lot of overhead and unnecessary bookkeeping, quickly turns out as useful information because it allows you to

Verify the origin and integrity of artifacts
Inspect the dependencies easily for CVEs and known security issues
Get a holistic view of your complete supply chain, so which other dependencies and Open Source projects now become part of your stack

Because it is structured data, all of the above can be done in an automated, programmatic fashion.

Big organizations, corporate or governmental, already keep track of SBOMs and make decisions based on the information provided there. Some started requiring SBOMs for software in-use. A good example of this is the government of the USA requiring SBOM from software suppliers.

Possible use-cases for SBOMs

Here are a couple more concrete examples of what the SBOMs for Flux allow you to do:

Security alerts of dependencies will be the most obvious use-case. If a CVE is detected, you can inspect your SBOM and see if the components you are using are in any way affected.
It will be easy to identify when a certain component was created and how. In addition to that, having the licensing information of all the pieces, you better know what you can do with it (redistribute, change, etc).
Automation allows you to send alerts, if a compliance issue is detected, e.g. if the licensing of updated/replaced dependencies changes.
Missing components or required build files. Incompatible licenses and more.

One example of automating all of this could be to store SBOMs in https://grafeas.io/. This way you could search across your Estate for:

Images that are built from a particular Github commit that is known to have introduced a security problem.
Find all images that were built by a certain version of a certain builder when that builder is known to have been compromised.
Find all images in my project that are impacted by CVE-1234.

For policy enforcement, kritis can be used to leverage the information provided by SBOMs inside Grafeas to enforce policies inside of a cluster, enabling auto-blocking of applications that are vulnerable to a specific CVE for example.

If you have read this far and you are using SBOMs in your organisation, let us know what you get out of them as well!

What’s more

If you would like to know more about the history of SBOMs and their development, you might want to read this excellent article from ChainGuard about the subject.

At the time of writing this, Syft does not yet classify licenses based on the file contents, but it is being considered.

Here is the table of SBOMs for all the latest Flux controllers and CLI (as of 2022-02-09).

Project	SBOM	Dependencies & Licenses
flagger	1.17.0	1.17.0
flux2	0.26.2	0.26.2
helm-controller	0.16.0	0.16.0
image-automation-controller	0.20.0	0.20.0
image-reflector-controller	0.16.0	0.16.0
kustomize-controller	0.20.1	0.20.1
notification-controller	0.21.0	0.21.0
source-controller	0.21.2	0.21.2

This is just one more measure we are taking to keep you more secure.

Talk to us

We love feedback, questions and ideas, so please let us know how you are using SBOMs today. Ask us if you have any questions and please

join our upcoming dev meetings
find us in the #flux channel on CNCF Slack
add yourself as an adopter if you haven’t already

See you around!

Blog: Flux Security Audit has concluded

Wed, 10 Nov 2021 12:30:00 +0000

As Flux is an Incubation project within the Cloud Native Computing Foundation, we were graciously granted a sponsored audit. The primary aim was to assess Flux’s fundamental security posture and to identify next steps in its security story. The audit was commissioned by the CNCF, and facilitated by OSTIF (the Open Source Technology Improvement Fund). ADA Logics was quickly brought into the picture, and spent a month on the audit.

The Flux maintainers and community are very grateful for the work put into this by everyone and the opportunity to grow and improve as a project.

Our first CVE in Flux

Let’s start with what will likely interest you as a Flux user. The engagement uncovered a privilege escalation vulnerability in Flux that could enable users to gain cluster admin privileges. The issue has been fixed and is assigned CVE 2021-41254, and the full disclosure advisory is available at the following link:

CVE-2021-41254: Privilege escalation to cluster admin on multi-tenant Flux.

Description:

Users that can create Kubernetes Secrets, Service Accounts and Flux Kustomization objects, could execute commands inside the kustomize-controller container by embedding a shell script in a Kubernetes Secret. This can be used to run kubectl commands under the Service Account of kustomize-controller, thus allowing an authenticated Kubernetes user to gain cluster admin privileges.

Impact:

Multi-tenant environments where non-admin users have permissions to create Flux Kustomization objects are affected by this issue.

Fix:

This vulnerability was fixed in kustomize-controller v0.15.0 (included in Flux v0.18.0) released on 2021-10-08. Starting with v0.15, the kustomize-controller no longer executes shell commands on the container OS and the kubectl binary has been removed from the container image.

Audit report with full details

We are thankful for the great attention to detail by the team at ADA Logics. The whole report can be found here. To benefit from the analysis in all its detail, we created a project board in GitHub. If you take a look at it closely, you will see that we have fixed some of the most immediate issues already.

Broadly speaking, the issues fall into three categories:

Enabling Fuzzing for the Flux project
Documentation issues
Concrete issues discovered in the Flux code

Flux coming to OSS-Fuzz

The team at ADA Logics didn’t stop at reviewing Flux code. We were pleasantly surprised to receive actual PRs by the team, who set down and helped us integrate with the OSS-Fuzz project. Some of this work still needs to be integrated into all of the Flux controllers, but we are very pleased that a start has been made! OSS-Fuzz is a service for running fuzzers continuously on important open source projects, and the goal is to use sophisticated dynamic analysis to uncover security and reliability issues. There are already numerous other CNCF projects integrated, e.g. Kubernetes, Envoy and Fluent-bit, and we’re excited to be a part of that.

Our documentation from an outside perspective

One very important piece of feedback was that our documentation is mostly geared towards end users, who need very concrete advice on how to integrate Flux into their setups. We provide lots of examples, which are helpful if you want Flux to behave the right way. What is missing to date is an architectural overview and documentation which focuses on the security-related aspects of Flux.

What transpired during the code review

The team at ADA Logics found 22 individual issues, some of which were results from the fuzzers. 1 high severity (that’s the above mentioned CVE), 3 medium severity, 13 low severity and 5 informational.

We appreciate the attention to detail by the team at ADA Logics. The issues range from dependency upgrades to oversights in the code (files which aren’t closed during an operation, unhandled errors) to misleading documentation.

Issue	Severity
1: Arbitrary command execution via command injection in the kustomize controller by way of secrets	High
2: Nil-dereference in image-automation controller	Low
3: Credentials exposed in environment variables and command line arguments	Medium
4: Use of deprecated library	Low
5: Invalid and missing testing documentation	Informational
6: Bug fixes do not always include regression tests	Informational
7: Deprecated SHA-1 is used for checksums	Low
8: Missing checksum verification	Medium
9: Inconsistent and missing logging	Low
10: Reading large files can crash flux with an out-of-memory bug	Low
11: Files are opened but never closed	Low
12: Unhandled error	Low
13: Slice bounds out of range	Low
14: Possible nil-deref in image-automation controller	Low
15: Inconsistent code-styles and potential nil-dereferences	Informational
16: Missing return statement after error	Low
17: File extension comparisons are case sensitive	Low
18: Some dependencies are outdated	Informational
19: Lack of container security options in deployed pods	Low
20: Unhandled errors from deferred file close operations	Low
21: x509 certificates are not used for Webex	Medium
22: Unnecessary conditions in the code	Informational

At the time of writing, 43% of the issues were still TODO, 21% WIP and 36% DONE.

The Road Ahead

We are very happy we were given the opportunity to work with and have our assumptions and code reviewed and tested by security experts. Early on we decided that we want to benefit from the findings as much as possible. That’s why we created a project board and added a review of it as a standing agenda item in our weekly dev meetings.

“The Flux team also created a public and easy to track dashboard showing all of the work we've done together and is a fantastic example of good issue-tracking and remediation.”

-- Derek Zimmer, President and Executive Director, OSTIF

Growing the team

If you are interested in contributing to this, we are very much looking forward to working with you. We welcome contributions in helping resolve issues of the road, additional comments on our security posture and also welcome contributions in the form of extending our fuzzing infrastructure. Finally, if you have any additional security feedback, please come and talk to us.

We are working full steam on the Flux Roadmap, just recently got more maintainers involved and continue to listen to feedback.

Again we would like to thank the Cloud Native Computing Foundation for sponsoring the audit, the Open Source Technology Improvement Fund for the coordination and ADA Logics for the careful review and advice during the audit period.

We are happy and proud to be part of this community!

Flux – security

Blog: Second Flux Security Audit has concluded

No new CVEs

Details on the discovered issues

Conclusion and next steps

Blog: Verify the integrity of the Helm Charts stored in OCI-compliant registries as OCI artifacts

DIY (Do it yourself) Approach

Blog: Prove the Authenticity of OCI Artifacts

Furthermore

Blog: May 2022 Security Announcement

tl;dr

Some Background

The advisories in detail

CVE-2022-24817 - Kubeconfig Validation

CVE-2022-24877 - Kustomization Path Traversal

CVE-2022-24878 - Kustomization Denial of Service

Inspecting your Flux system version

Vulnerable Flux system

Updating your vulnerable system

Up-to-date Flux system

Flux Security more generally speaking

What’s next for Flux

Blog: Security: Using Pod Security Standard "restricted"

Pod Security Standards definition

seccomp and RuntimeDefault

Further reading

Talk to us

Blog: Security: More confidence through Fuzzing

How we got here

What’s next

Talk to us

Blog: Security: Image Provenance

Why sign release artifacts in the first place?

How to verify signatures

Enforcing verified signatures in a cluster

Talk to us

Blog: Security: The Value of SBOMs

Flux - built with security in mind

What is a SBOM?

Possible use-cases for SBOMs

What’s more

Talk to us

Blog: Flux Security Audit has concluded

Our first CVE in Flux

Audit report with full details

Flux coming to OSS-Fuzz

Our documentation from an outside perspective

What transpired during the code review

The Road Ahead

Growing the team

`seccomp` and `RuntimeDefault`