Flux – Security Documentation

Flux: Security Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

The Flux project strives to keep its components secure by design and by default. This document aims to list all security-sensitive options or considerations that must be taken into account when deploying Flux. And also serve as a guide for security professionals auditing such deployments.

Not all recommendations are required for a secure deployment. Some may impact the convenience, performance or resources utilization of Flux. Therefore, use this in combination with your own Security Posture and Risk Appetite.

Some recommendations may overlap with Kubernetes security recommendations, to keep this short and more easily maintainable, please refer to Kubernetes CIS Benchmark for non Flux-specific guidance.

For help implementing these recommendations, seek enterprise support.

Security Best Practices

The recommendations below are based on Flux’s latest version.

Helm Controller

Start-up flags

Ensure controller was not started with --insecure-kubeconfig-exec=true.

Rationale

KubeConfigs support the execution of a binary command to return the token required to authenticate against a Kubernetes cluster.

This is very handy for acquiring contextual tokens that are time-bound (e.g. aws-iam-authenticator).
However, this may be open for abuse in multi-tenancy environments and therefore is disabled by default.
Audit Procedure

Check Helm Controller’s pod YAML for the arguments used at start-up:
```
kubectl describe pod -n flux-system -l app=helm-controller | grep -B 5 -A 10 Args
```
Ensure controller was not started with --insecure-kubeconfig-tls=true.

Rationale

Disables the enforcement of TLS when accessing the API Server of remote clusters.

This flag was created to enable scenarios in which non-production clusters need to be accessed via HTTP. Do not disable TLS in production.
Audit Procedure

Check Helm Controller’s pod YAML for the arguments used at start-up:
```
kubectl describe pod -n flux-system -l app=helm-controller | grep -B 5 -A 10 Args
```

Kustomize Controller

Start-up flags

Ensure controller was not started with --insecure-kubeconfig-exec=true.

Rationale

KubeConfigs support the execution of a binary command to return the token required to authenticate against a Kubernetes cluster.

This is very handy for acquiring contextual tokens that are time-bound (e.g. aws-iam-authenticator).

However, this may be open for abuse in multi-tenancy environments and therefore is disabled by default.
Audit Procedure

Check Kustomize Controller’s pod YAML for the arguments used at start-up:
```
kubectl describe pod -n flux-system -l app=kustomize-controller | grep -B 5 -A 10 Args
```
Ensure controller was not started with --insecure-kubeconfig-tls=true.

Rationale

Disables the enforcement of TLS when accessing the API Server of remote clusters.

This flag was created to enable scenarios in which non-production clusters need to be accessed via HTTP. Do not disable TLS in production.
Audit Procedure

Check Kustomize Controller’s pod YAML for the arguments used at start-up:
```
kubectl describe pod -n flux-system -l app=kustomize-controller | grep -B 5 -A 10 Args
```
Ensure controller was started with --no-remote-bases=true.

Rationale

By default the Kustomize controller allows for kustomize overlays to refer to external bases. This has a performance penalty, as the bases will have to be downloaded on demand during each reconciliation.
When using external bases, there can’t be any assurances that the externally declared state won’t change. In this case, the source loses its hermetic properties. Changes in the external bases will result in changes on the cluster, regardless of whether the source has been modified since the last reconciliation.
Audit Procedure

Check Kustomize Controller’s pod YAML for the arguments used at start-up:
```
kubectl describe pod -n flux-system -l app=kustomize-controller | grep -B 5 -A 10 Args
```

Secret Decryption

Ensure Secret Decryption is enabled and secrets are not being held in Flux Sources in plaintext.

Rationale

The kustomize-controller has an auto decryption mechanism that can decrypt cipher texts on-demand at reconciliation time using an embedded implementation of SOPS. This enables credentials (e.g. passwords, tokens) and sensitive information to be kept in an encrypted state in the sources.
Audit Procedure
- Check for plaintext credentials stored in the Git Repository at both HEAD and historical commits. Auto-detection tools can be used for this such as GitLeaks, Trufflehog and Squealer.
- Check whether Secret Decryption is properly enabled in each spec.decryption field of the cluster’s Kustomization objects.

Additional Best Practices for Shared Cluster Multi-tenancy

Multi-tenancy Lock-down

Ensure helm-controller, kustomize-controller, notification-controller, image-reflector-controller and image-automation-controller have cross namespace references disabled via --no-cross-namespace-refs=true.

Rationale

Blocks references to Flux objects across namespaces. This assumes that tenants would own one or multiple namespaces, and should not be allowed to consume other tenant’s objects, as this could enable them to gain access to sources they do not (or should not) have access to.
Audit Procedure

Check the Controller’s YAML for the arguments used at start-up:
```
kubectl describe pod -n flux-system -l app=helm-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=kustomize-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=notification-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=image-reflector-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=image-automation-controller | grep -B 5 -A 10 Args
```
Ensure helm-controller and kustomize-controller have a default service account set via --default-service-account=<service-account-name>.

Rationale

Enforces all reconciliations to impersonate a given Service Account, effectively disabling the use of the privileged service account that would otherwise be used by the controller.

Tenants must set a service account for each object that is responsible for applying changes to the Cluster (i.e. HelmRelease and Kustomization), otherwise Kubernetes’s API Server will not authorize the changes. NB: It is recommended that the default service account used has no permissions set to the control plane.
Audit Procedure

Check the Controller’s YAML for the arguments used at start-up:
```
kubectl describe pod -n flux-system -l app=helm-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=kustomize-controller | grep -B 5 -A 10 Args
```
Ensure all Flux controllers have default service accounts set for workload identity authentication via the --default-service-account=<service-account-name>, --default-decryption-service-account=<service-account-name> and --default-kubeconfig-service-account=<service-account-name> flags.

Rationale

In multi-tenant environments, workload identity authentication should be locked down to force tenant permissions used in cloud provider integrations to be provisioned following the Principle of Least Privilege. This ensures proper isolation between tenants with regards to ownership of cloud resources. This is separate from the Kubernetes RBAC impersonation controls mentioned above.

Setting default service accounts ensures that when Flux resources don’t specify a service account for workload identity authentication, they fall back to a controlled default expected to exist in the resource’s namespace, i.e. in the tenant’s namespace.

The workload identity default service account flags are --default-decryption-service-account and --default-kubeconfig-service-account for kustomize-controller, --default-kubeconfig-service-account for helm-controller, and --default-service-account for source-controller, notification-controller, image-reflector-controller and image-automation-controller.
Audit Procedure

Check all Flux controllers for workload identity default service account flags (--default-service-account, --default-decryption-service-account, --default-kubeconfig-service-account):
```
kubectl describe pod -n flux-system -l app=source-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=kustomize-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=helm-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=notification-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=image-reflector-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=image-automation-controller | grep -B 5 -A 10 Args
```

Secret Decryption

Ensure Secret Decryption is configured correctly, such that each tenant have the correct level of isolation.
Rationale

The secret decryption configuration must be aligned with the level of isolation required across tenants.
- For higher isolation, each tenant must have their own Key Encryption Key (KEK) configured. Note that the access controls to the aforementioned keys must also be aligned for better isolation.
- For lower isolation requirements, or for secrets that are shared across multiple tenants, cluster-level keys could be used.
Audit Procedure
- Check whether the Secret Provider configuration is security hardened. Please seek SOPS and SealedSecrets documentation for how to best implement each solution.
- When SealedSecrets are employed, pay special attention to the scopes being used.

Resource Isolation

Ensure additional Flux instances are deployed when mission-critical tenants/workloads must be assured.

Rationale

Sharing the same instances of Flux Components across all tenants including the Platform Admin, will lead to all reconciliations competing for the same resources. In addition, all Flux objects will be placed on the same queue for reconciliation which is limited by the number of workers set by each controller (i.e. --concurrent=20), which could cause reconciliation intervals not to be accurately honored.

For improved reliability, additional instances of Flux Components could be deployed, effectively creating separate “lanes” that are not disrupted by noisy neighbors. An example of this approach would be having additional instances of both Kustomize and Helm controllers that focuses on applying platform level changes, which do not compete with Tenants changes.

Running multiple Flux instances within the same cluster is supported by means of sharding, please consult the Flux sharding and horizontal scaling documentation for more details.

To avoid conflicts among controllers while attempting to reconcile Custom Resources, controller types (e.g. source-controller) must have be configured with unique label selectors in the --watch-label-selector flag.
Audit Procedure

Check for the existence of additional Flux controllers instances and their respective scopes. Each controller must be started with --watch-label-selector and have the selector point to unique label values:
```
kubectl describe pod -n flux-system -l app=kustomize-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=helm-controller | grep -B 5 -A 10 Args
kubectl describe pod -n flux-system -l app=source-controller | grep -B 5 -A 10 Args
```

Node Isolation

Ensure worker nodes are not being shared across tenants and the Flux components.

Rationale

Pods sharing the same worker node may enable threat vectors which might enable a malicious tenant to have a negative impact on the Confidentiality, Integrity or Availability of the co-located pods.

The Flux components may have Control Plane privileges while some tenants may not. A co-located pod could leverage its privileges in the shared worker node to bypass its own Control Plane access limitations by compromising one of the co-located Flux components. For cases in which cross-tenant isolation requirements must be enforced, the same risks apply.

Employ techniques to enforce that untrusted workloads are sandboxed. And, ensure that worker nodes are only shared when within the acceptable risks by your security requirements.
Audit Procedure
- Check whether you adhere to Kubernetes Node Isolation Guidelines
- Check whether there are Admission Controllers/OPA blocking tenants from creating privileged containers.
- Check whether RuntimeClass is being employed to sandbox workloads that may be scheduled in shared worker nodes.
- Check whether Taints and Tolerations are being used to decrease the likelihood of sharing worker nodes across tenants, or with the Flux controllers. Some cloud providers have this encapsulated as Node Pools.

Network Isolation

Ensure the Container Network Interface (CNI) being used in the cluster supports Network Policies.

Rationale

Flux relies on Network Policies to ensure that only Flux components have direct access to the source artifacts kept in the Source Controller.
Audit Procedure
- Check whether you adhere to Kubernetes Network Isolation Guidelines
- Confirm that the Network Policy objects created by Flux are being enforced by the CNI. Alternatively, run a tool such as Cyclonus or Sonobuoy to validate NetworkPolicy enforcement by the CNI plugin on your cluster.

Additional Best Practices for Tenant Dedicated Cluster Multi-tenancy

Ensure tenants are not able to revoke Platform Admin access to their clusters.

Rationale

In environments in which a management cluster is used to bootstrap and manage other clusters, it is important to ensure that a tenant is not allowed to revoke access from the Platform Admin, effectively denying the Management Cluster the ability to further reconcile changes into the tenant’s Cluster.

The Platform Admin should make sure that at the tenant’s cluster bootstrap process, this is taken into the account and a breakglass procedure is in place to recover access without the need to rebuild the cluster.
Audit Procedure
- Check whether alerts are in place in case the Remote Apply operations fails.
- Check the permission set given to the tenant’s users and applications is not overly privileged.
- Check whether there are Admission Controllers/OPA rules blocking changes in Platform Admin’s permissions and overall resources.

Flux: Contextual Authorization

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Most cloud providers support context-based authorization, enabling applications to benefit from strong access controls applied to a given context (e.g. Virtual Machine), without the need of managing authentication tokens and credentials.

For example, by granting a given Virtual Machine (or principal that such machine operates under) access to AWS S3, applications running inside that machine can request a token on-demand, which would grant them access to the AWS S3 buckets without having to store long lived credentials anywhere.

By leveraging such capability, Flux users can focus on the big picture, which is access controls enforcement with a least-privileged approach, whilst not having to do deal with security hygiene topics such as encrypting authentication secrets, and ensure they are being rotated regularly. All that is taken care of automatically by the cloud providers, as the tokens provided are context- and time-bound.

Current Support

Below is a list of Flux features that support this functionality and their documentation:

Status	Component	Feature	Provider	Ref
Supported	Source Controller	GitRepository Authentication	Azure	Guide
Supported	Source Controller	Bucket Authentication	AWS	Guide
Supported	Source Controller	Bucket Authentication	Azure	Guide
Supported	Source Controller	Bucket Authentication	GCP	Guide
Supported	Source Controller	OCIRepository Authentication	AWS	Guide
Supported	Source Controller	OCIRepository Authentication	Azure	Guide
Supported	Source Controller	OCIRepository Authentication	GCP	Guide
Supported	Source Controller	`oci` HelmRepository Authentication	AWS	Guide
Supported	Source Controller	`oci` HelmRepository Authentication	Azure	Guide
Supported	Source Controller	`oci` HelmRepository Authentication	GCP	Guide
Supported	Kustomize Controller	SOPS Integration with KMS	AWS	Guide
Supported	Kustomize Controller	SOPS Integration with Key Vault	Azure	Guide
Supported	Kustomize Controller	SOPS Integration with KMS	GCP	Guide
Supported	Kustomize Controller	Remote EKS Cluster Authentication	AWS	Guide
Supported	Kustomize Controller	Remote AKS Cluster Authentication	Azure	Guide
Supported	Kustomize Controller	Remote GKE Cluster Authentication	GCP	Guide
Supported	Helm Controller	Remote EKS Cluster Authentication	AWS	Guide
Supported	Helm Controller	Remote AKS Cluster Authentication	Azure	Guide
Supported	Helm Controller	Remote GKE Cluster Authentication	GCP	Guide
Supported	Notification Controller	Azure DevOps Commit Status Updates	Azure	Guide
Supported	Notification Controller	Azure Event Hubs	Azure	Guide
Supported	Notification Controller	Google Cloud Pub/Sub	GCP	Guide
Supported	Image Reflector Controller	ImageRepository Authentication	AWS	Guide
Supported	Image Reflector Controller	ImageRepository Authentication	Azure	Guide
Supported	Image Reflector Controller	ImageRepository Authentication	GCP	Guide
Supported	Image Automation Controller	GitRepository Authentication	Azure	Guide

Flux: Secrets Management

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Flux improves the application deployment process by continuously reconciling a desired state, defined at a source, against a target cluster. One of the challenges with this process is its dependency on secrets, that must not be stored in plain-text like the rest of the desired state.

Secrets are sensitive information that an application needs to operate such as: credentials, passwords, keys, tokens and certificates. Managing secrets declaratively needs to be done right because of its broad security implications.

We will cover the mechanisms supported by Flux, as well as the security principles, concerns and techniques to consider when managing secrets with Flux.

What’s inside the toolbox?

First of all, let’s go through the different options supported by Flux and Kubernetes.

Nowadays there are a multitude of secret management options. Some are available in-cluster, directly in the comfort of your Kubernetes cluster, and others that are provided from out-of-cluster, for example a Cloud based KMS.

Kubernetes Secrets

Kubernetes has a built-in mechanism to store and manage secrets. The secrets are stored in etcd either in plain-text or encrypted.

They are the vanilla offering, which is used during flux bootstrap, for example, to store your SSH Deploy Keys. Unless when the initial Flux source supports contextual authorization, in which case no secrets are required.

Storing plain-text secrets in your desired state is not recommended, so apart from the secret used to authenticate against your initial source, Flux users should not manage these. Instead, they should rely mostly on other mechanisms covered below.

Secrets Decryption Operators

Sometimes referred to as Encrypted Secrets, Secrets Decryption Operators enable secrets to be stored in ciphertext as Kubernetes resources within a Flux source. They are deployed into the cluster by Flux in their original CustomResourceDefinition (CRD) form, which is later used by its Secret Decryption Operator to decrypt those secrets and generate a Kubernetes Secret.

This is transparent to the consuming applications, making it a quite suitable approach to retrofit into an existing setup. An example of a Secret Decryption Operator is Sealed Secrets.

Storing encrypted secrets in Git repositories enables configuration versioning to leverage the same practices used for managing, versioning and releasing application and/or infrastructure when using declarative “everything as code”, for example pull-requests, tags and branch strategies.

Note that some sources may keep a history of the encrypted Secrets (e.g. GitRepository) through time. Increasing the impact when old encryption keys are leaked, especially when other security measures are not in place (e.g. secret rotation) or when long-lived secrets are being handled (e.g. Public TLS Certs).

Notice that secrets can be stored in any Source type supported by Flux, such as Buckets and OCI repositories.

Flux specific guides on using Secrets Decryption Operators:

Bitnami Sealed Secrets

Using Flux to decrypt Secrets on-demand

Flux has the ability to decrypt secrets stored in Flux sources by itself, without the need of additional controllers installed in the cluster. The approach relies on keeping in Flux sources encrypted Kubernetes Secrets, which are decrypted on-demand with SOPS, just before they are deployed into the target clusters.

This approach is more flexible than using Sealed Secrets, as SOPS supports cloud-based Key Management Services of the major cloud providers (Azure KeyVault, GCP KMS and AWS KMS), HashiCorp Vault, as well as “off-line” decryption using Age and PGP.

This mechanism supports kustomize-secretgenerator which ensures that dependent workloads will reload automatically and start using the latest version of the secret. Notice that most approaches that are based on Kubernetes Secrets would require something like stakater/Reloader to achieve the same result. The Kubernetes blog explains quite well how this works.

The security concerns of this approach are similar to the Secrets Decryption Operators, but with the added benefit that no additional controllers are required, therefore reducing resources consumption and the attack surface. When using external providers (e.g. KMS, Vault), remember that they can become a single point of failure, if they are deleted by mistake (or unavailable by extended periods) this could impact your solution.

Flux supports the two main names in Encrypted Secrets and has specific how-to guides for them:

Secrets Synchronized by Operators

The source of truth for your secrets can reside outside of the cluster, and then be synchronised into the cluster as Kubernetes Secrets by operators. Much like encrypted secrets, this process is transparent to the workloads in the cluster.

Two examples of this type of operator are 1Password Operator and External Secrets Operator. But given their nature, Flux is able to support any operator that manages Kubernetes secrets.

This approach provides a level of redundancy by default, as secrets are kept at both the cluster and the remote source, so small failures can go undetected. It supports hybrid workloads quite well, when some secrets have to be shared with applications that are not Kubernetes-based.

When using mutable secrets, it could be hard for Flux or the dependent applications to know whether they are using the latest version of a given secret. In such cases, immutable secrets, where the name also contains the version of the secret, may help.

Take into account the loading times when provisioning a new cluster, as that can become a bottleneck slowing down the provisioning time as the number of secrets increases.

Flux supports all operators that provide this functionality.

Secrets mounted via CSI Drivers

Another way to bring external secrets into Kubernetes, is the use of CSI Drivers, which mounts secrets as files directly into a Pod filesystem, instead of generating native Kubernetes Secrets.

Due to the way it works, the secrets are not accessible within the Kubernetes Control Plane, so although you can use it with your workloads, it won’t work when providing to CustomResourceTypes (CRDs) that need a reference to a secret (e.g. .spec.secretRef.name in GitRepository).

With CSI Drivers, the mounting takes place at Pod starting time, so issues accessing the external source of the secrets may be more impactful.

Here are a few CSI providers:

Direct access to out-of-cluster Secrets

Direct access to a secret management solution that resides outside of a Kubernetes cluster is also an option. Which could be a useful alternative when lifting and shifting legacy applications that already depend on such approach.

Here the secret management solution will become a single point of failure, expect issues when it goes temporarily unavailable and make sure to have disaster recovery plans. Also observe throttling limits of cloud solutions, given that different applications may be targeting the same Secret Manager, without a rate limiter across all of them, this could easily lead to an outage at scale.

Flux currently does not directly fetch secrets from out-of-cluster solutions, in the same way that most Kubernetes native tools don’t, therefore this approach may need to be combined with things such as Secrets Synchronized by Operators. However, this will not block the ability of your applications to do so.

Big Picture - Things to consider

Once you are aware of the different tools in the toolbox, it is important to align them with your actual requirements, taking into account some key points:

Expiration and Rotation

Secrets should have an expiration time, and ideally such expiration should be enforced, so that the potential of leakage has a well-defined risk window.

To facilitate the uninterrupted use of the dependent applications, rotation should be automated taking into account that at times different versions of the same secret (old and new) may need to be supported at the same time - e.g. whilst validating a new version of the application that is being deployed.

Both secrets can remain active during a time window, but once the new version is validated after deployment, the previous secrets can be safely decommissioned. Cloud KMS solutions tend to provide secret versioning built-in.

Access Management and Auditing

Access to secrets should be restricted to the servers and applications within the environment they need to be accessed. The same goes for users and service accounts.

When considering the different solutions, it is important to note how they hang together and what the gaps are. If you have a strong requirement for access and auditing controls, having a well-defined api-server auditing in place, together with tight RBAC policies in your cluster is only part of the problem. Also take into account how those secrets are sourced, stored and handled. Maybe having secrets stored (even if in encrypted form) in an easily accessible Flux source that has a loosely defined RBAC and no auditing in place may not meet such requirements.

Least Privileged and Segregation of duties

The scope of each secret must be carefully considered to decrease the blast radius in case of breach. A trade-off must be reached to attain a balance between the two extremes: having a single secret that has all the access, versus having too many secrets that are always used in combination.

Sharing the same secret across different scopes, just because they have the same permissions may lead to disruption if such secret needs to be quickly rotated.

Disaster Recovery

The entire provisioning of your infrastructure and application must take into account break the glass procedures that are secure, provide relevant security controls (e.g. auditing) and cannot be misused to bypass other processes (e.g. Access Management).

Around disaster recovery scenarios, consider how they align with your Availability and Confidentiality requirements.

Don’t co-locate ciphertext with encryption keys

It should go without saying, but never place secrets together with keys that can provide privilege escalation routes. For example, if you store the decryption key for your secrets in GitHub secrets, and all your encrypted secrets are stored in the same repository, a single GitHub account (with enough access) compromised is enough for all your secrets to be decrypted.

Instead, segregate encryption keys from ciphertext and understand what needs to be compromised for the data to be at risk.

Single Points of Failure

Identify all potential single points of failure and ensure that there is a way around them. If all your secrets are encrypted using an encryption key stored in Vault, and due to a major failure your Vault instance is completely lost, and no backup is to be found, the encrypted secrets are now useless. Therefore, think big picture, and ensure that each step of the way has a redundancy and that process is regularly exercised.

The same goes for temporary single points of failure. If you rely on a Key Hierarchy Architecture based on a cloud KMS to provision an on-premises cluster/application, consider the impact they would have in case of a failure pre, mid or post deploy (of either cluster or applications).

Ephemeral or Single-use Secrets

The easiest type of secrets to manage are the ones that ephemeral; context-bound and time-bound. However, they are not supported by all use-cases. Whenever they are, prioritise their use over static or long-lived secrets.

An example of an ephemeral secret that is time-bound, is a token provided by cloud providers to any application running within a given Cloud Machine. Those tokens are generated automatically, and have a short expiration time. In some cases you can even tie them to a network boundary, meaning that even if they get breached, they won’t be able to be used outside the current context.

Flux supports contextual authorization for the major Cloud Providers, be aware of the supported features and use them whenever possible.

Detect “chicken and egg” scenarios

Flux won’t protect you from yourself. On a running cluster, it is quite easy to incrementally fall into the trap of building a non-provisionable cluster. For example, if your first Kustomization depends on a CustomResourceType (CRD) to deploy a secret, which is only deployed as part of another Kustomization, Flux may not be able to redeploy your sources from scratch on a new cluster.

Make sure that your pipeline identifies and tests such scenarios. Automate the provisioning of clusters that can test the entire E2E of your deployment process, and ensure that it is executed regularly.

Summary

Flux supports a wide range of Secret Management solutions. And it is up to its users to define what works best for their use case. This subject isn’t easy, and due diligence is important to ensure the appropriate level of security controls are in place.

Overall, none of the approaches covered above are inherently secure or insecure, but they are rather part of a big picture in which what matters the most is the weakest link and how it all hangs together. As with all things around security, a layered approach is recommended.

Take into account your threat model, availability and resilience requirements when deciding what works best for you, and rest assured that a combination of some of the above will make more sense, especially when disaster recovery and break the glass scenarios are considered.

Flux: SLSA Assessment

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Supply Chain Levels for Software Artifacts, or SLSA (pronounced “salsa”), is a security framework which aims to prevent tampering and secure artifacts in a project. SLSA is designed to support automation that tracks code handling from source to binary protecting against tampering regardless of the complexity of the software supply chain.

Starting with Flux version 2.0.0, the build, release and provenance portions of the Flux project supply chain provisionally meet SLSA Build Level 3.

SLSA Requirements and Flux Compliance State

What follows is an assessment made by members of the Flux core maintainers team on how Flux v2.0 complies with the Build Level 3 requirements as specified by SLSA v1.0.

Producer Requirements

Requirement	Required at SLSA L3	Met by Flux
Choose an appropriate build platform	Yes	Yes
Follow a consistent build process	Yes	Yes
Distribute provenance	Yes	Yes

Choose an appropriate build platform

The producer MUST choose a builder capable of producing Build Level 3 provenance.

The Flux project uses Git for source code management and the Flux project’s repositories are hosted on GitHub under the FluxCD organization.
All the Flux maintainers are required to have two-factor authentication enabled and to sign-off all their contributions.
The Flux project uses GitHub Actions and GitHub Runners for building all its release artifacts.
The build and release process runs in isolation on an ephemeral environment provided by GitHub-hosted runners.

Follow a consistent build process

The producer MUST build their artifact in a consistent manner such that verifiers can form expectations about the build process.

The build and release process is defined in code (GitHub Workflows and Makefiles) and is kept under version control.
The GitHub Workflows make use of GitHub Actions pinned to their Git commit SHA and are kept up-to-date using GitHub Dependabot.
All changes to build and release process are done via Pull Requests that must be approved by at least one Flux maintainer.
The release process can only be kicked off by a Flux maintainer by pushing a Git tag in the semver format.

Distribute provenance

The producer MUST distribute provenance to artifact consumers.

The Flux project uses the official SLSA GitHub Generator project for provenance generation and distribution.
The provenance for the release artifacts published to GitHub releases (binaries, SBOMs, deploy manifests, source code) is generated using the generator_generic_slsa3 GitHub Workflow provided by the SLSA GitHub Generator project.
The provenance for the release artifacts published to GitHub Container Registry and to DockerHub (Flux controllers multi-arch container images) is generated using the generator_container_slsa3 GitHub Workflow provided by the SLSA GitHub Generator project.

Build Platform Requirements

Provenance generation

Requirement	Required at SLSA L3	Met by Flux
Provenance Exists	Yes	Yes
Provenance is Authentic	Yes	Yes
Provenance is Unforgeable	Yes	Yes

The build process MUST generate provenance that unambiguously identifies the output package by cryptographic digest and describes how that package was produced.

The Flux project release workflows make use of the official SLSA GitHub Generator project for provenance generation.
The provenance file stores the SHA-256 hashes of the release artifacts (binaries, SBOMs, deploy manifests, source code).
The provenance identifies the Flux container images using their digest in SHA-256 format.

Consumers MUST be able to validate the authenticity of the provenance attestation in order to ensure integrity and define trust.

The provenance is signed by Sigstore Cosign using the GitHub OIDC identity and the public key to verify the provenance is stored in the public Rekor transparency log.
The release process and the provenance generation are run in isolation on an ephemeral environment provided by GitHub-hosted runners.
The provenance of the Flux release artifacts (binaries, container images, SBOMs, deploy manifests) can be verified using the official SLSA verifier tool.

Provenance MUST be strongly resistant to forgery by tenants.

The provenance generation workflows run on ephemeral and isolated virtual machines which are fully managed by GitHub.
The provenance signing secrets are ephemeral and are generated through Sigstore’s keyless signing procedure.
The SLSA GitHub generator runs on separate virtual machines than the build and release process, so that the Flux build scripts don’t have access to the signing secrets.

Isolation strength

Requirement	Required at SLSA L3	Met by Flux
Hosted	Yes	Yes
Isolated	Yes	Yes

All build steps ran using a hosted build platform on shared or dedicated infrastructure.

The release process and the provenance generation are run in isolation on an ephemeral environment provided by GitHub-hosted runners.
The provenance generation is decoupled from the build process, the SLSA GitHub generator runs on separate virtual machines fully managed by GitHub.

The build platform ensured that the build steps ran in an isolated environment, free of unintended external influence.

The release process can only be kicked off by a Flux maintainer by pushing a Git tag in the semver format.
The release process runs on ephemeral and isolated virtual machines which are fully managed by GitHub.
The release process can’t access the provenance signing key, because the provenance generator runs in isolation on separate GitHub-hosted runners.

Provenance verification

The provenance of the Flux release artifacts (binaries, container images, SBOMs, deploy manifests) can be verified using the official SLSA verifier tool.

Container images

The provenance of the Flux multi-arch container images hosted on GitHub Container Registry and DockerHub can be verified using the official SLSA verifier tool and Sigstore Cosign.

What follows is the list of Flux components along with their minimum required version for provenance verification.

Git Repository	Images	Min version	Provenance (SLSA L3)
flux2	`docker.io/fluxcd/flux-cli` `ghcr.io/fluxcd/flux-cli`	`v2.0.1`	Yes
source-controller	`docker.io/fluxcd/source-contoller` `ghcr.io/fluxcd/source-contoller`	`v1.0.0`	Yes
kustomize-controller	`docker.io/fluxcd/kustomize-contoller` `ghcr.io/fluxcd/kustomize-contoller`	`v1.0.0`	Yes
notification-controller	`docker.io/fluxcd/notification-contoller` `ghcr.io/fluxcd/notification-contoller`	`v1.0.0`	Yes
helm-controller	`docker.io/fluxcd/helm-contoller` `ghcr.io/fluxcd/helm-contoller`	`v0.35.0`	Yes
image-reflector-controller	`docker.io/fluxcd/image-reflector-contoller` `ghcr.io/fluxcd/image-reflector-contoller`	`v0.29.0`	Yes
image-automation-controller	`docker.io/fluxcd/image-automation-contoller` `ghcr.io/fluxcd/image-automation-contoller`	`v0.35.0`	Yes

Example

We will be using the source-controller container image hosted on GHCR for this example, but these instructions can be used for all Flux container images.

First, collect the digest of the image to verify:

$ crane digest ghcr.io/fluxcd/source-controller:v1.0.0
sha256:8dfd386a338eab2fde70cd7609e3b35a6e2f30283ecf2366da53013295fa65f3

Using the digest, verify the provenance of the Flux controller by specifying the repository and version:

$ slsa-verifier verify-image ghcr.io/fluxcd/source-controller:v1.0.0@sha256:8dfd386a338eab2fde70cd7609e3b35a6e2f30283ecf2366da53013295fa65f3 --source-uri github.com/fluxcd/source-controller --source-tag v1.0.0
Verified build using builder https://github.comslsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.7.0 at commit a40e0da705f26710077a7591f9dad05b7cd55acd
PASSED: Verified SLSA provenance

Using Cosign, verify the SLSA provenance attestation by specifying the workflow and GitHub OIDC issuer:

$ cosign verify-attestation --type slsaprovenance --certificate-identity-regexp https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v --certificate-oidc-issuer https://token.actions.githubusercontent.com ghcr.io/fluxcd/source-controller:v1.0.0
Verification for ghcr.io/fluxcd/source-controller:v1.0.0 --
The following checks were performed on each of these signatures:
 - The cosign claims were validated
 - Existence of the claims in the transparency log was verified offline
 - The code-signing certificate was verified using trusted certificate authority certificates
Certificate subject: https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.7.0
Certificate issuer URL: https://token.actions.githubusercontent.com
GitHub Workflow Trigger: push
GitHub Workflow SHA: a40e0da705f26710077a7591f9dad05b7cd55acd
GitHub Workflow Name: release
GitHub Workflow Repository: fluxcd/source-controller
GitHub Workflow Ref: refs/tags/v1.0.0

Flux artifacts

The provenance of the Flux release artifacts (binaries, container images, SBOMs, deploy manifests) published on GitHub can be verified using the official SLSA verifier tool.

Example

In this example we use the Flux SBOM file, but the instructions can be used for all artifacts included with the Flux release.

First, download the release artifacts from GitHub:

FLUX_VER=2.0.1 && \
gh release download v${FLUX_VER} -R=fluxcd/flux2 -p="*"

Using the provenance.intoto.jsonl file, verify the provenance attestation of the Flux SBOM (flux_<version>_sbom.spdx.json):

$ slsa-verifier verify-artifact --provenance-path provenance.intoto.jsonl --source-uri github.com/fluxcd/flux2 --source-tag v${FLUX_VER} flux_${FLUX_VER}_sbom.spdx.json
Verified signature against tlog entry index 27066821 at URL: https://rekor.sigstore.dev/api/v1/log/entries/24296fb24b8ad77ac2d2dc6381ec7f1f04d991344771214c5fb5861621dbd9da6f0551f806cbf609
Verified build using builder https://github.comslsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.7.0 at commit 9b3162495ce1b99b1fcdf137c553f543eafe3ec7
Verifying artifact flux_2.0.1_sbom.spdx.json: PASSED
PASSED: Verified SLSA provenance