Flux – Flux configuration

Flux: Flux optional components

Mon, 01 Jan 0001 00:00:00 +0000

The flux bootstrap command deploys a series of Kubernetes controllers along with their CRDs, RBAC and network policies in the flux-system namespace.

Default components

The default components are specified with the --components flag.

flux bootstrap git \
 --components source-controller,kustomize-controller,helm-controller,notification-controller

The minimum required components for bootstrapping are source-controller and kustomize-controller.

When not specifying the --components flag, both flux bootstrap and flux install will deploy the default components.

Extra components

To enable the Flux image automation feature, the extra components can be specified with the --components-extra flag.

flux bootstrap git \
 --components-extra image-reflector-controller,image-automation-controller

To enable the Flux source composition and decomposition features, the source-watcher component can be added to the extra components.

flux bootstrap git \
 --components-extra source-watcher

By default, both flux bootstrap and flux install commands do not include any extra components.

Network policies

Flux relies on Kubernetes network policies to ensure that only Flux components have direct access to the source artifacts kept in the source-controller.

The default network policies block all ingress access to the flux-system namespace, except for notification-controller webhook receiver.

While not recommend, you can deploy the Flux components without network policies using the --network-policy flag.

flux bootstrap git \
 --network-policy false

Cluster domain

The Flux components are communicating over HTTP with source-controller and notification-controller. To reduce the DNS queries performed by each component, the cluster internal domain name is used to compose the FQDN of each service e.g. http://source-controller.flux-system.svc.cluster.local./.

The cluster domain name can be set using the --cluster-domain flag.

flux bootstrap git \
 --cluster-domain cluster.internal

When not specified, the cluster domain defaults to cluster.local.

Flux: Flux bootstrap customization

Mon, 01 Jan 0001 00:00:00 +0000

To customize the Flux controllers during bootstrap, first you’ll need to create a Git repository and clone it locally.

Create the file structure required by bootstrap with:

mkdir -p clusters/my-cluster/flux-system
touch clusters/my-cluster/flux-system/gotk-components.yaml \
 clusters/my-cluster/flux-system/gotk-sync.yaml \
 clusters/my-cluster/flux-system/kustomization.yaml

The Flux controller deployments, container command arguments, node affinity, etc can be customized using Kustomize strategic merge patches and JSON patches.

You can make changes to all controllers using a single patch or target a specific controller:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: # manifests generated during bootstrap
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 # target all controllers
 - patch: | 
 # strategic merge or JSON patch
 target:
 kind: Deployment
 labelSelector: "app.kubernetes.io/part-of=flux"
 # target controllers by name
 - patch: |
 # strategic merge or JSON patch
 target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller)"
 # add a command argument to a single controller
 - patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=5
 target:
 kind: Deployment
 name: "source-controller"

Push the changes to main branch:

git add -A && git commit -m "init flux" && git push

And run the bootstrap for clusters/my-cluster:

flux bootstrap git \
 --url=ssh://git@<host>/<org>/<repository> \
 --branch=main \
 --path=clusters/my-cluster

To make further amendments, pull the changes locally, edit the kustomization.yaml file, push the changes upstream and rerun bootstrap.

Flux: Flux vertical scaling

Mon, 01 Jan 0001 00:00:00 +0000

When Flux is managing hundreds of applications that are deployed multiple times per day, cluster admins can fine tune the Flux controller at bootstrap time to run at scale by:

Increasing the number of workers and resource limits
Enabling in-memory kustomize builds
Enabling Helm repository caching to reduce memory usage
Enabling persistent storage for internal artifacts
Running the Flux controllers on dedicated nodes

Horizontal scaling

When vertical scaling is not an option, you can use sharding to horizontally scale the Flux controllers. For more details please see the sharding guide.

Increase the number of workers and limits

If Flux is managing hundreds of applications, it is advised to increase the number of reconciliations that can be performed in parallel and to bump the resources limits:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s
 target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller|source-controller)"
 - patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: all
 spec:
 template:
 spec:
 containers:
 - name: manager
 resources:
 limits:
 cpu: 2000m
 memory: 2Gi
 target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller|source-controller)"

I/O Device Contention

Note that further increasing the number of concurrent reconciliations for kustomize-controller, without using a RAM-backed filesystem, may not have the desired effect due to IO contention on the Kubernetes node disk.

Enable in-memory kustomize builds

When increasing the number of concurrent reconciliations, it is advised to use tmpfs for the /tmp filesystem to speed up the Flux kustomize build operations:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=20
 - op: replace
 path: /spec/template/spec/volumes/0
 value:
 name: temp
 emptyDir:
 medium: Memory
 target:
 kind: Deployment
 name: kustomize-controller

Memory usage

Note that tmpfs will count against the controller’s pod memory usage, so it is advised to make use of the GitRepository ignore feature to exclude non-YAML files when Flux syncs app repos.

Enable Helm repositories caching

If Flux connects to Helm repositories hosting hundreds of Helm charts, it is advised to enable caching to reduce the memory footprint of source-controller:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --helm-cache-max-size=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --helm-cache-ttl=60m
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --helm-cache-purge-interval=5m
 target:
 kind: Deployment
 name: source-controller

When helm-cache-max-size is reached, an error is logged and the index is instead read from file. Cache hits are exposed via the gotk_cache_events_total Prometheus metrics. Use this data to fine-tune the configuration flags.

Persistent storage for Flux internal artifacts

Flux maintains a local cache of artifacts acquired from external sources. By default, the cache is stored in an EmptyDir volume, which means that after a restart, Flux has to restore the local cache by fetching the content of all Git repositories, Buckets, Helm charts and OCI artifacts. To avoid losing the cached artifacts, you can configure source-controller with a persistent volume.

Create a Kubernetes PVC definition named gotk-pvc.yaml and place it in your flux-system directory:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gotk-pvc
 namespace: flux-system
spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: standard
 resources:
 requests:
 storage: 10Gi

Add the PVC file to the kustomization.yaml resources and patch the source-controller volumes:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
 - gotk-pvc.yaml
patches:
 - patch: |
 - op: add
 path: '/spec/template/spec/volumes/-'
 value:
 name: persistent-data
 persistentVolumeClaim:
 claimName: gotk-pvc
 - op: replace
 path: '/spec/template/spec/containers/0/volumeMounts/0'
 value:
 name: persistent-data
 mountPath: /data
 target:
 kind: Deployment
 name: source-controller

Node affinity and tolerations

Pin the Flux controller pods to specific nodes and allow the cluster autoscaler to evict them:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: all
 spec:
 template:
 metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 spec:
 affinity:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: role
 operator: In
 values:
 - flux
 tolerations:
 - effect: NoSchedule
 key: role
 operator: Equal
 value: flux
 target:
 kind: Deployment
 labelSelector: app.kubernetes.io/part-of=flux

The above configuration pins Flux to nodes tainted and labeled with:

kubectl taint nodes <my-node> role=flux:NoSchedule
kubectl label nodes <my-node> role=flux

Flux: Flux sharding and horizontal scaling

Mon, 01 Jan 0001 00:00:00 +0000

When Flux is managing tens of thousands of applications, it is advised to adopt a sharding strategy to spread the load between multiple instances of Flux controllers. To enable horizontal scaling, each controller can be deployed multiple times with a unique label selector which is used as the sharding key.

What follows is a guide on how to bootstrap multiple controller instances and how to shard the reconciliation of Flux resources using the sharding.fluxcd.io/key label.

Bootstrap with sharding

At bootstrap time, you can define the number of shards and spin up a Flux controller instance for each shard.

First you’ll need to create a Git repository and clone it locally, then create the file structure required by bootstrap with:

mkdir -p clusters/my-cluster/flux-system
touch clusters/my-cluster/flux-system/gotk-components.yaml \
 clusters/my-cluster/flux-system/gotk-sync.yaml \
 clusters/my-cluster/flux-system/kustomization.yaml

Then you’ll create a dedicated directory inside flux-system for each shard:

mkdir -p clusters/my-cluster/flux-system/shard1
touch clusters/my-cluster/flux-system/shard1/kustomization.yaml

Configure controller sharding

In the shard1 directory generate a set of controller deployments that will reconcile the Flux resources labels with sharding.fluxcd.io/key: shard1.

To spin up a dedicated source-controller, kustomize-controller and helm-controller instance, use the following patches in clusters/my-cluster/flux-system/shard1/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system
resources:
 - ../gotk-components.yaml
nameSuffix: "-shard1"
commonAnnotations:
 sharding.fluxcd.io/role: "shard"
patches:
 - target:
 kind: (Namespace|CustomResourceDefinition|ClusterRole|ClusterRoleBinding|ServiceAccount|NetworkPolicy|ResourceQuota)
 labelSelector: "app.kubernetes.io/part-of=flux"
 patch: |
 apiVersion: v1
 kind: all
 metadata:
 name: all
 $patch: delete
 - target:
 labelSelector: "app.kubernetes.io/component=notification-controller"
 patch: |
 apiVersion: v1
 kind: all
 metadata:
 name: all
 $patch: delete
 - target:
 kind: Deployment
 name: (image-reflector-controller|image-automation-controller)
 patch: |
 apiVersion: v1
 kind: Deployment
 metadata:
 name: all
 $patch: delete
 - target:
 kind: Service
 name: source-controller
 patch: |
 - op: replace
 path: /spec/selector/app
 value: source-controller-shard1
 - target:
 kind: Deployment
 name: source-controller
 patch: |
 - op: replace
 path: /spec/selector/matchLabels/app
 value: source-controller-shard1
 - op: replace
 path: /spec/template/metadata/labels/app
 value: source-controller-shard1
 - op: replace
 path: /spec/template/spec/containers/0/args/6
 value: --storage-adv-addr=source-controller-shard1.$(RUNTIME_NAMESPACE).svc.cluster.local.
 - target:
 kind: Deployment
 name: kustomize-controller
 patch: |
 - op: replace
 path: /spec/selector/matchLabels/app
 value: kustomize-controller-shard1
 - op: replace
 path: /spec/template/metadata/labels/app
 value: kustomize-controller-shard1
 - target:
 kind: Deployment
 name: helm-controller
 patch: |
 - op: replace
 path: /spec/selector/matchLabels/app
 value: helm-controller-shard1
 - op: replace
 path: /spec/template/metadata/labels/app
 value: helm-controller-shard1
 - target:
 kind: Deployment
 name: (source-controller|kustomize-controller|helm-controller)
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --watch-label-selector=sharding.fluxcd.io/key=shard1

The above configuration will generate three deployments source-controller-shard1, kustomize-controller-shard1 and helm-controller-shard1 all configured with --watch-label-selector=sharding.fluxcd.io/key=shard1.

To enable these deployments at bootstrap, add the shard1 directory to the clusters/my-cluster/flux-system/kustomization.yaml resources:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
- shard1
patches:
 - target:
 kind: Deployment
 name: "(source-controller|kustomize-controller|helm-controller)"
 annotationSelector: "!sharding.fluxcd.io/role"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/0
 value: --watch-label-selector=!sharding.fluxcd.io/key

Note how this configuration excludes the sharding keys from the main controllers watch with --watch-label-selector=!sharding.fluxcd.io/key. This ensures that the main controllers will not reconcile any Flux resources labels with the sharding keys.

Install and Upgrade shards

Push the changes to main branch:

git add -A && git commit -m "init flux" && git push

And run the bootstrap for clusters/my-cluster:

flux bootstrap git \
 --url=ssh://git@<host>/<org>/<repository> \
 --branch=main \
 --path=clusters/my-cluster

Verify that the main controllers and the ones assigned to shard1 are running:

$ kubectl -n flux-system get deployments

NAME READY UP-TO-DATE AVAILABLE AGE
helm-controller 1/1 1 1 1m
helm-controller-shard1 1/1 1 1 1m
kustomize-controller 1/1 1 1 1m
kustomize-controller-shard1 1/1 1 1 1m
notification-controller 1/1 1 1 1m
source-controller 1/1 1 1 1m
source-controller-shard1 1/1 1 1 1m

When upgrading Flux, either by rerunning bootstrap with a newer version or by using the Flux GitHub Actions, the sharded controllers will be automatically upgraded along with the main ones.

Assign resources to shards

To assign a group of Flux resources to a particular shard, label them with sharding.fluxcd.io/key.

For example, assuming you want to assign the reconciliation of an application to the shard1 controllers, label both the Flux source and its Kustomization with sharding.fluxcd.io/key: shard1:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: podinfo
 namespace: default
 labels:
 sharding.fluxcd.io/key: shard1
spec:
 interval: 10m
 url: https://github.com/stefanprodan/podinfo
 ref:
 semver: 6.x
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: podinfo
 namespace: default
 labels:
 sharding.fluxcd.io/key: shard1
spec:
 interval: 10m
 targetNamespace: default
 sourceRef:
 kind: GitRepository
 name: podinfo
 path: ./kustomize
 prune: true

Note that Source object kinds which have a dependency on another kind (i.e. HelmChart on a HelmRepository) need to have the same labels applied to work as expected.

For example, assuming you want to assign the reconciliation of a Helm release to the shard1 controllers, label the HelmRelease, its chart and its repository with sharding.fluxcd.io/key: shard1:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: podinfo
 namespace: default
 labels:
 sharding.fluxcd.io/key: shard1
spec:
 interval: 10m
 type: oci
 url: oci://ghcr.io/stefanprodan/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: default
 labels:
 sharding.fluxcd.io/key: shard1
spec:
 interval: 10m
 chart:
 metadata:
 labels:
 sharding.fluxcd.io/key: shard1
 spec:
 chart: podinfo
 sourceRef:
 kind: HelmRepository
 name: podinfo

Note that with .spec.chart.metadata.labels we set the sharding key on the generated Flux HelmChart object, so that both the Helm repository and charts are managed by source-controller-shard1 instance.

Bulk assign shards

Instead of manually labeling each Flux resource with a shard key, use a top-level Flux Kustomization and automatically label all resources.

For example, assuming you want to assign a tenant to a particular shard, in the root Flux Kustomization that reconcile the tenant’s Flux sources, kustomizations and Helm releases label these resources as follows:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: tenant1
 namespace: tenant1
spec:
 interval: 10m0s
 path: ./apps
 prune: true
 sourceRef:
 kind: GitRepository
 name: tenant1
 targetNamespace: tenant1
 commonMetadata:
 labels:
 sharding.fluxcd.io/key: shard1
 patches:
 - target:
 kind: HelmRelease
 patch: |
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 metadata:
 name: all
 spec:
 chart:
 metadata:
 labels:
 sharding.fluxcd.io/key: shard1

With .spec.commonMetadata.labels we set the sharding key on all the Flux resources and with the .spec.patches we set the same sharding key for all generated HelmCharts.

Flux: Flux multi-tenancy

Mon, 01 Jan 0001 00:00:00 +0000

Flux allows different organizations and/or teams to share the same Kubernetes control plane; this is referred to as “multi-tenancy”. To make this safe, Flux supports segmentation and isolation of resources by using namespaces and role-based access control (RBAC).

Flux authorization model

Flux defers to Kubernetes’ native RBAC to specify which operations are authorized when processing its custom resources. By default, this means operations are constrained by the service account under which the controllers run, which has the cluster-admin role bound to it. This is convenient for a deployment in which all users are trusted.

In a multi-tenant deployment, each tenant needs to be restricted in the operations that can be done on their behalf. Since tenants control Flux via its API objects, this becomes a matter of attaching RBAC rules to Flux API objects.

To give users control over the authorization, the Flux controllers can impersonate (assume the identity of) a service account mentioned in the apply specification (e.g., the field .spec.serviceAccountName in a Kustomization object or in a HelmRelease object) for both accessing resources and applying configuration. This lets a user constrain the operations performed by the Flux controllers with RBAC.

Flux user roles

The tenancy model assume two types of users: platform admins and tenants. Besides installing Flux, all the other operations (deploy applications, configure ingress, policies, etc) do not require users to have direct access to the Kubernetes API. Flux acts as a proxy between users and the Kubernetes API, using Git and OCI as the source of truth for the cluster desired state.

Platform Admins

The platform admins have unrestricted access to Kubernetes API. They are responsible for installing Flux and granting Flux access to the sources (Git, Helm, OCI repositories) that make up the cluster(s) control plane desired state. The repository(s) owned by the platform admins are reconciled on the cluster(s) by Flux, under the cluster-admin Kubernetes cluster role.

Example of operations performed by platform admins:

Bootstrap Flux onto cluster(s).
Extend the Kubernetes API with custom resource definitions and validation webhooks.
Configure various controllers for ingress, storage, logging, monitoring, progressive delivery, etc.
Set up namespaces for tenants and define their level of access with Kubernetes RBAC.
Onboard tenants by registering their Git repositories with Flux.

Tenants

The tenants have restricted access to the cluster(s) according to the Kubernetes RBAC configured by the platform admins. The repositories owned by tenants are reconciled on the cluster(s) by Flux, under the Kubernetes account(s) assigned by platform admins.

Example of operations performed by tenants:

Register their sources with Flux (GitRepositories, HelmRepositories and Buckets).
Deploy workload(s) into their namespace(s) using Flux custom resources (Kustomizations and HelmReleases).
Automate application updates using Flux custom resources (ImageRepositories, ImagePolicies and ImageUpdateAutomations).
Configure the release pipeline(s) using Flagger custom resources (Canaries and MetricsTemplates).
Setup webhooks and alerting for their release pipeline(s) using Flux custom resources (Receivers and Alerts).

How to configure Flux multi-tenancy

A platform admin can lock down Flux on multi-tenant clusters during bootstrap with the following patches:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --no-cross-namespace-refs=true
 target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
 - patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --no-remote-bases=true
 target:
 kind: Deployment
 name: "kustomize-controller"
 - patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --default-service-account=default
 target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller)"
 - patch: |
 - op: add
 path: /spec/serviceAccountName
 value: kustomize-controller
 target:
 kind: Kustomization
 name: "flux-system"

With the above configuration, Flux will:

Deny cross-namespace access to Flux custom resources, thus ensuring that a tenant can’t use another tenant’s sources or subscribe to their events.
Deny accesses to Kustomize remote bases, thus ensuring all resources refer to local files, meaning only the Flux Sources can affect the cluster-state.
All Kustomizations and HelmReleases which don’t have spec.serviceAccountName specified, will use the default account from the tenant’s namespace. Tenants have to specify a service account in their Flux resources to be able to deploy workloads in their namespaces as the default account has no permissions.
The flux-system Kustomization is set to reconcile under a service account with cluster-admin role, allowing platform admins to configure cluster-wide resources and provision the tenant’s namespaces, service accounts and RBAC.

Multi-tenancy example

The Flux team maintains a dedicated repository at github.com/fluxcd/flux2-multi-tenancy to showcase how platform admins can onboard tenants and limit their access to the cluster resources with various policies.

Flux cluster role aggregations

By default, Flux RBAC grants Kubernetes builtin view, edit and admin roles access to Flux custom resources.

This allows tenants to manage Flux resources in their own namespaces using a Service Account with a role binding to admin.

If you wish to disable the RBAC aggregation, you can remove the flux-view and flux-edit cluster roles with:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
 name: flux
 $patch: delete
 target:
 kind: ClusterRole
 name: "(flux-view|flux-edit)-flux-system"

Flux Object-level Workload Identity RBAC

Starting with v2.6, Flux supports object-level workload identity, which requires additional RBAC permissions to be granted to the controllers so that they can create ServiceAccount tokens:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: crd-controller
rules:
 # excerpt from the existing rules
 - apiGroups:
 - ""
 resources:
 - serviceaccounts/token
 verbs:
 - create

If you wish to disable the object-level workload identity RBAC in Flux 2.6 or later, you can do so with the following patch:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 - op: remove
 path: /rules/10
 target:
 kind: ClusterRole
 name: "crd-controller-flux-system"

Flux: Flux Workload Identity

Mon, 01 Jan 0001 00:00:00 +0000

When running Flux on AWS EKS, Azure AKS and Google Cloud GKE you can leverage Kubernetes Workload Identity to grant Flux controllers access to cloud resources such as container registries, KMS, S3, etc.

AWS IAM roles for service accounts

To grant Flux access to AWS resources using IRSA during bootstrap add the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 eks.amazonaws.com/role-arn: <ECR ROLE ARN>
 target:
 kind: ServiceAccount
 name: "(source-controller|image-reflector-controller)"
 - patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 eks.amazonaws.com/role-arn: <KMS ROLE ARN>
 target:
 kind: ServiceAccount
 name: "kustomize-controller"

Amazon Web Services Integrations

For a complete guide on configuring authentication for the Flux integrations with AWS IAM for accessing ECR, S3 and KMS please see the AWS integration documentation.

Azure Workload Identity

To grant Flux access to Azure resources during bootstrap add the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 azure.workload.identity/client-id: <AZURE CLIENT ID>
 azure.workload.identity/tenant-id: <AZURE TENANT ID>
 target:
 kind: ServiceAccount
 name: "(source-controller|image-reflector-controller)"
 - patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: controller
 labels:
 azure.workload.identity/use: "true"
 spec:
 template:
 metadata:
 labels:
 azure.workload.identity/use: "true" 
 target:
 kind: Deployment
 name: "(source-controller|image-reflector-controller)"

Microsoft Azure Integrations

For a complete guide on configuring authentication for the Flux integrations with Microsoft Entra ID for accessing ACR, Azure DevOps, Blob Storage, Key Vault and Event Hubs please see the Azure integration documentation.

GCP Workload Identity

To grant Flux access to Google Cloud resources during bootstrap add the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 iam.gke.io/gcp-service-account: <GAR ID NAME>
 target:
 kind: ServiceAccount
 name: "(source-controller|image-reflector-controller)"
 - patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 iam.gke.io/gcp-service-account: <KMS ID NAME>
 target:
 kind: ServiceAccount
 name: "kustomize-controller"

Google Cloud Platform Integrations

For a complete guide on configuring authentication for the Flux integrations with GCP IAM for accessing GAR, GCS, KMS and Pub/Sub please see the GCP integration documentation.

Flux: Flux air-gapped installation

Mon, 01 Jan 0001 00:00:00 +0000

Flux can be installed on air-gapped environments where the Kubernetes cluster, the container registry and the Git server are not connected to the internet.

Copy the container images

On a machine with access to github.com and ghcr.io, download the Flux CLI from GitHub releases page.

List the Flux container images with:

flux install --export | grep ghcr.io

Copy each controller images to your private container registry using crane:

FLUX_CONTROLLERS=(
"source-controller"
"kustomize-controller"
"helm-controller"
"notification-controller"
"image-reflector-controller"
"image-automation-controller"
)

for controller in "${FLUX_CONTROLLERS[@]}"; do
 crane copy --all-tags ghcr.io/fluxcd/$controller <your-registry>/$controller
done

Bootstrap Flux

Copy the Flux CLI binary to the machine inside the air-gapped network and run bootstrap using the images from your private registry:

flux bootstrap git \
 --registry=registry.internal/fluxcd \
 --url=ssh://git@<host>/<org>/<repository> \
 --branch=<my-branch> \
 --private-key-file=<path/to/private.key> \
 --password=<key-passphrase> \
 --path=clusters/my-cluster

Note that you must generate a SSH private key and set the public key as the deploy key on your Git server in advance.

For more information on how to use the flux bootstrap git command, please see the generic Git server documentation.

Bootstrap Flux and authenticate to a private container registry

If the private container registry requires authentication, you can pass the credentials to the Flux CLI using the --registry-creds flag. Replace username:password with your credentials and run:

flux bootstrap git \
 --registry-creds=username:password \
 --image-pull-secret=regcred \
 --registry=registry.internal/fluxcd \
 --url=ssh://git@<host>/<org>/<repository> \
 --branch=<my-branch> \
 --private-key-file=<path/to/private.key> \
 --password=<key-passphrase> \
 --path=clusters/my-cluster

The Flux CLI creates the image pull secret specified with --image-pull-secret in the flux-system namespace and configures the controllers pods to use it.

To update the credentials in the image pull secret, you can re-run the bootstrap command with the new credentials or update the secret directly with:

flux -n flux-system create secret oci regcred \
 --username=username \
 --password=password

Flux: Flux OpenShift installation

Mon, 01 Jan 0001 00:00:00 +0000

Required permissions

To bootstrap Flux, the person running the command must have cluster admin rights for the target OpenShift cluster. It is also required to prepare a Git repository as described in the bootstrap customization.

First copy the scc.yaml to the flux-system directory. This manifest contains the RBAC necessary to allow the Flux controllers to run as non-root on OpenShift.

Then add the scc.yaml and the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
 - scc.yaml
patches:
 - patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: all
 spec:
 template:
 spec:
 securityContext:
 $patch: delete
 containers:
 - name: manager
 securityContext:
 runAsUser: 65534
 seccompProfile:
 $patch: delete
 target:
 kind: Deployment
 labelSelector: app.kubernetes.io/part-of=flux
 - patch: |-
 - op: remove
 path: /metadata/labels/pod-security.kubernetes.io~1warn
 - op: remove
 path: /metadata/labels/pod-security.kubernetes.io~1warn-version
 target:
 kind: Namespace
 labelSelector: app.kubernetes.io/part-of=flux

Finally, push the changes to the Git repository and run flux bootstrap.

OperatorHub

Flux can be installed on Red Hat OpenShift cluster directly from OperatorHub using Flux Operator.

The Flux Operator is an open-source project part of the Flux ecosystem that provides a declarative API for the lifecycle management of the Flux controllers on OpenShift.

First create a Subscription resource in the flux-system namespace to install the Flux Operator:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: flux-operator
 namespace: flux-system
spec:
 channel: stable
 name: flux-operator
 source: operatorhubio-catalog
 sourceNamespace: olm
 config:
 env:
 - name: DEFAULT_SERVICE_ACCOUNT
 value: "flux-operator"

After the subscription, create a FluxInstance resource with .spec.cluster.type set to openshift:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
 name: flux
 namespace: flux-system
 annotations:
 fluxcd.controlplane.io/reconcileEvery: "1h"
spec:
 distribution:
 version: "2.x"
 registry: "ghcr.io/fluxcd"
 components:
 - source-controller
 - kustomize-controller
 - helm-controller
 - notification-controller
 - image-reflector-controller
 - image-automation-controller
 cluster:
 type: openshift
 multitenant: true
 networkPolicy: true
 domain: "cluster.local"
 sync:
 kind: GitRepository
 url: "https://my-git-server.com/my-org/my-fleet.git"
 ref: "refs/heads/main"
 path: "clusters/my-cluster"
 pullSecret: "flux-system"

For more information on how to configure the Flux instance, refer to the following resources:

Flux: Flux proxy settings

Mon, 01 Jan 0001 00:00:00 +0000

If the Kubernetes cluster has Internet restrictions, requiring egress traffic to go through a proxy for accessing external services, you can configure the Flux controllers to use a HTTP/S and/or SOCKS5 SSH proxy.

Using HTTP/S proxy for egress traffic

If your cluster uses an HTTP proxy to reach GitHub or other external services, you must set NO_PROXY=.cluster.local.,.cluster.local,.svc to allow the Flux controllers to talk to each other.

To set the HTTP/S proxy during bootstrap, add the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: all
 spec:
 template:
 spec:
 containers:
 - name: manager
 env:
 - name: "HTTPS_PROXY"
 value: "https://proxy.example.com"
 - name: "NO_PROXY"
 value: ".cluster.local.,.cluster.local,.svc"
 target:
 kind: Deployment
 labelSelector: app.kubernetes.io/part-of=flux

Git repository access via SOCKS5 SSH proxy

To configure a SOCKS5 proxy, set the environment variable ALL_PROXY to allow both source-controller and image-automation-controller to connect through the proxy.

To set the SOCKS5 proxy during bootstrap, add the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: all
 spec:
 template:
 spec:
 containers:
 - name: manager
 env:
 - name: "ALL_PROXY"
 value: "socks5://proxy.example.com:1080"
 target:
 kind: Deployment
 labelSelector: app.kubernetes.io/part-of=flux
 name: "(source-controller|image-automation-controller)"

Flux: Flux deploy key rotation

Mon, 01 Jan 0001 00:00:00 +0000

There are several reasons you may want to rotate the deploy key:

The token used to generate the key has expired.
The key has been compromised.
You want to change the scope of the key, e.g. to allow write access using the --read-write-key flag to flux bootstrap.

While you can run flux bootstrap repeatedly, be aware that the flux-system Kubernetes Secret is never overwritten. You need to manually rotate the key as described here.

To rotate the SSH key generated at bootstrap, first delete the secret from the cluster with:

kubectl -n flux-system delete secret flux-system

Then you have two alternatives to generate a new key:

Generate a new secret with
```
flux create secret git flux-system \
 --url=ssh://git@<host>/<org>/<repository>
```
The above command will print the SSH public key, once you set it as the deploy key, Flux will resume all operations.
Run flux bootstrap ... again. This will generate a new key pair and, depending on which Git provider you use, print the SSH public key that you then set as deploy key or automatically set the deploy key (e.g. with GitHub).

Flux: Flux drift detection for Helm Releases

Mon, 01 Jan 0001 00:00:00 +0000

Since Flux v2.2.0, drift detection for Helm releases is available as a configuration option on the HelmRelease itself. Refer to the HelmRelease documentation for more information.

Flux: Flux near OOM detection for Helm

Mon, 01 Jan 0001 00:00:00 +0000

When memory usage of the helm-controller exceeds the configured limit, the controller will forcefully be killed by Kubernetes’ OOM killer. This may result in a Helm release being left in a pending-* state which causes the HelmRelease to get stuck in an another operation (install/upgrade/rollback) is in progress error loop.

To prevent this from happening, the controller offers an OOM watcher which can be enabled with --feature-gates=OOMWatch=true. When enabled, the memory usage of the controller will be monitored, and a graceful shutdown will be triggered when it reaches a certain threshold (default 95% utilization).

When gracefully shutting down, running Helm actions may mark the release as failed. Because of this, enabling this feature is best combined with thoughtful remediation strategies.

To enable near OOM detection during bootstrap add the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 # Enable OOM watch feature
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --feature-gates=OOMWatch=true
 # Threshold at which to trigger a graceful shutdown (optional, default 95%)
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --oom-watch-memory-threshold=95
 # Interval at which to check memory usage (optional, default 500ms)
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --oom-watch-interval=500ms
 target:
 kind: Deployment
 name: helm-controller

Flux: Flux DNS lookups for Helm Releases

Mon, 01 Jan 0001 00:00:00 +0000

By default, the helm-controller will not perform DNS lookups when rendering Helm templates in clusters because of potential security implications.

To enable DNS lookups during bootstrap add the following patches to the flux-system kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 # Allow Helm DNS lookups
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --feature-gates=AllowDNSLookups=true
 target:
 kind: Deployment
 name: helm-controller