Flux – Migrate from Flux v1

Flux: Migrate from Flux v1 to v2

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through migrating from Flux v1 to v2. Read the FAQ to find out what differences are between v1 and v2.

Prerequisites

You will need a Kubernetes cluster version 1.20 or newer and kubectl version 1.20 or newer.

Install Flux v2 CLI

With Homebrew:

brew install fluxcd/tap/flux

With Bash:

curl -s https://fluxcd.io/install.sh | sudo bash

# enable completions in ~/.bash_profile
. <(flux completion bash)

Command-line completion for zsh, fish, and powershell are also supported with their own sub-commands.

Binaries for macOS, Windows and Linux AMD64/ARM are available for download on the release page.

Verify that your cluster satisfies the prerequisites with:

flux check --pre

GitOps migration

Flux v2 offers an installation procedure that is declarative first and disaster resilient.

Using the flux bootstrap command you can install Flux on a Kubernetes cluster and configure it to manage itself from a Git repository. The Git repository created during bootstrap can be used to define the state of your fleet of Kubernetes clusters.

For a detailed walk-through of the bootstrap procedure please see the installation guide.

'flux bootstrap' target

flux bootstrap should not be run against a Git branch or path that is already being synchronized by Flux v1, as this will make them fight over the resources. Instead, bootstrap to a new Git repository, branch or path, and continue with moving the manifests.

After you’ve installed Flux v2 on your cluster using bootstrap, you can delete the Flux v1 from your clusters and move the manifests from the Flux v1 repository to the bootstrap one. Typically deleting Flux v1 can be done by deleting these helm installations: flux and helm-operator

One key change in Flux v2 is “server-side apply” that enforces strict validation of the manifests. It is important to note that manifests are applied atomically to the cluster only if server side validation of the apply passes. If one Kustomization has multiple resources, an error in any one of the resources will also prevent other resources in that group from getting applied. This is a breaking change from Flux v1.

In-place migration

For production use we recommend using the bootstrap procedure (see the Gitops migration section above), but if you wish to install Flux v2 in the same way as Flux v1 then follow along.

Flux read-only mode

Assuming you’ve installed Flux v1 to sync a directory with plain YAMLs from a private Git repo:

# create namespace
kubectl create ns flux

# deploy Flux v1
fluxctl install \
--git-url=git@github.com:org/app \
--git-branch=main \
--git-path=./deploy \
--git-readonly \
--namespace=flux | kubectl apply -f -

# print deploy key
fluxctl identity --k8s-fwd-ns flux

# trigger sync
fluxctl sync --k8s-fwd-ns flux

Uninstall Flux v1

Before you proceed, scale the Flux v1 deployment to zero or delete its namespace and RBAC.

If there are YAML files in your deploy dir that are not meant to be applied on the cluster, you can exclude them by placing a .sourceignore in your repo root:

$ cat .sourceignore
# exclude all
/*
# include deploy dir
!/deploy
# exclude files from deploy dir
/deploy/**/eksctl.yaml
/deploy/**/charts

Install Flux v2 in the flux-system namespace:

$ flux install \
 --network-policy=true \
 --watch-all-namespaces=true \
 --namespace=flux-system
✚ generating manifests
✔ manifests build completed
► installing components in flux-system namespace
✔ install completed
◎ verifying installation
✔ source-controller ready
✔ kustomize-controller ready
✔ helm-controller ready
✔ notification-controller ready
✔ install finished

$ flux create source git app \
 --url=ssh://git@github.com/org/app \
 --branch=main \
 --interval=1m
► generating deploy key pair
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCp2x9ghVmv1zD...
Have you added the deploy key to your repository: y
► collecting preferred public key from SSH server
✔ collected public key from SSH server:
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A...
► applying secret with keys
✔ authentication configured
✚ generating GitRepository source
► applying GitRepository source
✔ GitRepository source created
◎ waiting for GitRepository source reconciliation
✔ GitRepository source reconciliation completed
✔ fetched revision: main@sha1:5302d04c2ab8f0579500747efa0fe7abc72c8f9b

Configure the reconciliation of the deploy dir on your cluster:

$ flux create kustomization app \
 --source=GitRepository/app \
 --path="./deploy" \
 --prune=true \
 --interval=10m
✚ generating Kustomization
► applying Kustomization
✔ Kustomization created
◎ waiting for Kustomization reconciliation
✔ Kustomization app is ready
✔ applied revision main@sha1:5302d04c2ab8f0579500747efa0fe7abc72c8f9b

If your repository contains secrets encrypted with Mozilla SOPS, please read this guide.

Pull changes from Git and apply them immediately:

flux reconcile kustomization app --with-source

List all Kubernetes objects reconciled by app:

kubectl get all --all-namespaces \
-l=kustomize.toolkit.fluxcd.io/name=app \
-l=kustomize.toolkit.fluxcd.io/namespace=flux-system

Flux with Kustomize

Assuming you’ve installed Flux v1 to sync a Kustomize overlay from an HTTPS Git repository:

fluxctl install \
--git-url=https://github.com/org/app \
--git-branch=main \
--manifest-generation \
--namespace=flux | kubectl apply -f -

With the following .flux.yaml in the root dir:

version: 1
patchUpdated:
 generators:
 - command: kustomize build ./overlays/prod
 patchFile: flux-patch.yaml

Uninstall Flux v1

Before you proceed, delete the Flux v1 namespace and remove the .flux.yaml from your repo.

Install Flux v2 in the flux-system namespace:

flux install

flux create source git app \
 --url=https://github.com/org/app \
 --branch=main \
 --username=git \
 --password=token \
 --interval=1m

Configure the reconciliation of the prod overlay on your cluster:

flux create kustomization app \
 --source=GitRepository/app \
 --path="./overlays/prod" \
 --prune=true \
 --interval=10m

Check the status of the Kustomization reconciliation:

$ flux get kustomizations app
NAME REVISION SUSPENDED READY
app main@sha1:5302d04c2ab8f0579500747efa0fe7abc72c8f9b False True

Flux with Slack notifications

Assuming you have configured Flux v1 to send notifications to Slack with FluxCloud.

With Flux v2, create an alert provider for a Slack channel:

flux create alert-provider slack \
 --type=slack \
 --channel=some-channel-name \
 --address=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK

And configure notifications for the app reconciliation events:

flux create alert app \
 --provider-ref=slack \
 --event-severity=info \
 --event-source=GitRepository/app \
 --event-source=Kustomization/app

For more details, read the guides on how to configure notifications and webhooks.

Flux debugging

Check the status of Git operations:

$ kubectl -n flux-system get gitrepositories
NAME READY MESSAGE
app True Fetched revision: main@sha1:5302d04c2ab8f0579500747efa0fe7abc72c8f9b
test False SSH handshake failed: unable to authenticate, attempted methods [none publickey]

Check the status of the cluster reconciliation with kubectl:

$ kubectl -n flux-system get kustomizations
NAME READY STATUS
app True Applied revision: main@sha1:5302d04c2ab8f0579500747efa0fe7abc72c8f9
test False The Service 'backend' is invalid: spec.type: Unsupported value: 'Ingress'

Suspend a reconciliation:

$ flux suspend kustomization app
► suspending kustomization app in flux-system namespace
✔ kustomization suspended

Check the status with kubectl:

$ kubectl -n flux-system get kustomization app
NAME READY STATUS
app False Kustomization is suspended, skipping reconciliation

Resume a reconciliation:

$ flux resume kustomization app
► resuming Kustomization app in flux-system namespace
✔ Kustomization resumed
◎ waiting for Kustomization reconciliation
✔ Kustomization reconciliation completed
✔ applied revision main@sha1:5302d04c2ab8f0579500747efa0fe7abc72c8f9b

Flux: Migrating image update automation to Flux v2

Mon, 01 Jan 0001 00:00:00 +0000

“Image Update Automation” is a process in which Flux makes commits to your Git repository when it detects that there is a new image to be used in a workload (e.g., a Deployment). In Flux v2 this works quite differently to how it worked in Flux v1. This guide explains the differences and how to port your cluster configuration from v1 to v2. There is also a tutorial for using image update automation with a new cluster.

Overview of changes between v1 and v2

In Flux v1, image update automation (from here, just “automation”) was built into the Flux daemon, which scanned everything it found in the cluster and updated the Git repository it was syncing.

In Flux v2,

automation is controlled with custom resources, not annotations
ordering images by build time is not supported (there is a section below explaining what to do instead)
the fields to update in files are marked explicitly, rather than inferred from annotations.

Automation is now controlled by custom resources

Flux v2 breaks down the functions in Flux v1’s daemon into controllers, with each having a specific area of concern. Automation is now done by two controllers: one which scans image repositories to find the latest images, and one which uses that information to commit changes to git repositories. These are in turn separate to the syncing controllers.

This means that automation in Flux v2 is governed by custom resources. In Flux v1 the daemon scanned everything, and looked at annotations on the resources to determine what to update. Automation in v2 is more explicit than in v1 – you have to mention exactly which images you want to be scanned, and which fields you want to be updated.

A consequence of using custom resources is that with Flux v2 you can have an arbitrary number of automations, targeting different Git repositories if you wish, and updating different sets of images. If you run a multitenant cluster, the tenants can define automation in their own namespaces, for their own Git repositories.

Selecting an image is more flexible

The ways in which you choose to select an image have changed. In Flux v1, you generally supply a filter pattern, and the latest image is the image with the most recent build time out of those filtered. In Flux v2, you choose an ordering, and separately specify a filter for the tags to consider. These are dealt with in detail below.

Selecting an image by build time is no longer supported. This is the implicit default in Flux v1. In Flux v2, you will need to tag images so that they sort in the order you would like – see below for how to do this conveniently.

Fields to update are explicitly marked

Lastly, in Flux v2 the fields to update in files are marked explicitly. In Flux v1 they are inferred from the type of the resource, along with the annotations given. The approach in Flux v1 was limited to the types that had been programmed in, whereas Flux v2 can update any Kubernetes object (and some files that aren’t Kubernetes objects, like kustomization.yaml).

Preparing for migration

It is best to complete migration of your system to Flux v2 syncing first, using the Flux v1 migration guide. This will remove Flux v1 from the system, along with its image automation. You can then reintroduce automation with Flux v2 by following the instructions in this guide.

It is safe to leave the annotations for Flux v1 in files while you reintroduce automation, because Flux v2 will ignore them.

To migrate to Flux v2 automation, you will need to do three things:

make sure you are running the automation controllers; then,
declare the automation with an ImageUpdateAutomation object; and,
migrate each manifest by translate Flux v1 annotations to Flux v2 ImageRepository and ImagePolicy objects, and putting update markers in the manifest file.

Where to keep `ImageRepository`, `ImagePolicy` and `ImageUpdateAutomation` manifests

This guide assumes you want to manage automation itself via Flux. In the following sections, manifests for the objects controlling automation are saved in files, committed to Git, and applied in the cluster with Flux.

A Flux v2 installation will typically have a Git repository structured like this:

<...>/flux-system/
gotk-components.yaml
gotk-sync.yaml
<...>/app/
# deployments etc.

The <...> is the path to a particular cluster’s definitions – this may be simply ., or something like clusters/my-cluster. To get the files in the right place, set a variable for this path:

$ CLUSTER_PATH=<...> # e.g., "." or "clusters/my-cluster", or ...
$ AUTO_PATH=$CLUSTER_PATH/automation
$ mkdir ./$AUTO_PATH

The file $CLUSTER_PATH/flux-system/gotk-components.yaml has definitions of all the Flux v2 controllers and custom resource definitions. The file gotk-sync.yaml defines a GitRepository and a Kustomization which will sync manifests under $CLUSTER_PATH/.

To these will be added definitions for automation objects. This guide puts manifest files for automation in $CLUSTER_PATH/automation/, but there is no particular structure required by Flux. The automation objects do not have to be in the same namespace as the objects to be updated.

Migration on a branch

This guide assumes you will commit changes to the branch that is synced by Flux, as this is the simplest way to understand.

It may be less disruptive to put migration changes on a branch, then merging when you have completed the migration. You would need to either change the GitRepository to point at the migration branch, or have separate GitRepository and Kustomization objects for the migrated parts of your Git repository. The main thing to avoid is syncing the same objects in two different places; e.g., avoid having Kustomizations that sync both the unmigrated and migrated application configuration.

Installing the command-line tool `flux`

The command-line tool flux will be used below; see these instructions for how to install it.

Running the automation controllers

The first thing to do is to deploy the automation controllers to your cluster. The best way to proceed will depend on the approach you took when following the Flux read-only migration guide.

If you used flux bootstrap to create a new Git repository, then ported your cluster configuration to that repository, use After flux bootstrap;
If you used flux install to install the controllers directly, use After migrating Flux v1 in place;
If you used flux install and exported the configuration to a file, use After committing Flux v2 configuration to Git.

After `flux bootstrap`

When starting from scratch, you are likely to have used flux bootstrap. Rerun the command, and include the image automation controllers in your starting configuration with the flag --components-extra, as shown in the installation guide.

This will commit changes to your Git repository and sync them in the cluster.

flux check --components-extra=image-reflector-controller,image-automation-controller

Now jump to the section Migrating each manifest to Flux v2.

After migrating Flux v1 in place

If you followed the Flux v1 migration guide, you will already be running some Flux v2 controllers. The automation controllers are currently considered an optional extra to those, but are installed and run in much the same way. You may or may not have committed the Flux v2 configuration to your Git repository. If you did, go to the section After committing Flux v2 configuration to Git.

If not, you will be installing directly to the cluster:

$ flux install --components-extra=image-reflector-controller,image-automation-controller

It is safe to repeat the installation command, or to run it after using flux bootstrap, so long as you repeat any arguments you supplied the first time.

Now jump ahead to Migrating each manifest to Flux v2.

After committing a Flux v2 configuration to Git

If you added the Flux v2 configuration to your git repository, assuming it’s in the file $CLUSTER_PATH/flux-system/gotk-components.yaml as used in the guide, use flux install and write it back to that file:

$ flux install \
 --components-extra=image-reflector-controller,image-automation-controller \
 --export > "$CLUSTER_PATH/flux-system/gotk-components.yaml"

Commit changes to the $CLUSTER_PATH/flux-system/gotk-components.yaml file and sync the cluster:

$ git add $CLUSTER_PATH/flux-system/gotk-components.yaml
$ git commit -s -m "Add image automation controllers to Flux config"
$ git push
$ flux reconcile kustomization --with-source flux-system

Controlling automation with an `ImageUpdateAutomation` object

In Flux v1, automation was run by default. With Flux v2, you have to explicitly tell the controller which Git repository to update and how to do so. These are defined in an ImageUpdateAutomation object; but first, you need a GitRepository with write access, for the automation to use.

If you followed the Flux v1 read-only migration guide, you will have a GitRepository defined in the namespace flux-system, for syncing to use. This GitRepository will have read access to the Git repository by default, and automation needs write access to push commits.

To give it write access, you can replace the secret it refers to. How to do this will depend on what kind of authentication you used to install Flux v2.

Replacing the Git credentials secret

The secret with Git credentials will be named in the .spec.secretRef.name field of the GitRepository object. Say your GitRepository is in the namespace flux-system and named flux-system (these are the defaults if you used flux bootstrap); you can retrieve the secret name and Git URL with:

$ FLUX_NS=flux-system
$ GIT_NAME=flux-system
$ SECRET_NAME=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath={.spec.secretRef.name})
$ GIT_URL=$(kubectl -n $FLUX_NS get gitrepository $GIT_NAME -o jsonpath='{.spec.url}')
$ echo $SECRET_NAME $GIT_URL # make sure they have values

If you’re not sure which kind of credentials you’re using, look at the secret:

$ kubectl -n $FLUX_NS describe secret $SECRET_NAME

An entry at .data.identity indicates that you are using an SSH key (the first section below); an entry at .data.username indicates you are using a username and password or token (the second section below).

Replacing an SSH key secret

When using an SSH (deploy) key, create a new key:

$ flux create secret git -n $FLUX_NS $SECRET_NAME --url=$GIT_URL

You will need to copy the public key that’s printed out, and install that as a deploy key for your Git repo making sure to check the ‘All write access’ box (or otherwise give the key write permissions). Remove the old deploy key.

Replacing a username/password secret

When you’re using a username and password to authenticate, you may be able to change the permissions associated with that account.

If not, you will need to create a new access token (e.g., “Personal Access Token” in GitHub). In this case, once you have the new token you can replace the secret with the following:

$ flux create secret git -n $FLUX_NS $SECRET_NAME \
 --username <username> --password <token> --url $GIT_URL

Checking the new credentials

To check if your replaced credentials still work, try syncing the GitRepository object:

$ flux reconcile source git -n $FLUX_NS $GIT_NAME
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ GitRepository reconciliation completed
✔ fetched revision main@sha1:d537304e8f5f41f1584ca1e807df5b5752b2577e

When this is successful, it tells you the new credentials have at least read access.

Making an automation object

To set automation running, you create an ImageUpdateAutomation object. Each object will update a Git repository, according to the image policies in the namespace.

Here is an ImageUpdateAutomation manifest for the example (note: you will have to supply your own value for at least the host part of the email address):

$ # the environment variables $AUTO_PATH and $GIT_NAME are set above
$ FLUXBOT_EMAIL=fluxbot@example.com # supply your own host or address here
$ flux create image update my-app-auto \
 --author-name FluxBot --author-email "$FLUXBOT_EMAIL" \
 --git-repo-ref $GIT_NAME \
 --checkout-branch main \
 --push-branch main \
 --interval 30m \
 --export > ./$AUTO_PATH/my-app-auto.yaml
$ cat my-app-auto.yaml
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: my-app-auto
 namespace: flux-system
spec:
 interval: 30m
 sourceRef:
 kind: GitRepository
 name: flux-system
 git:
 checkout:
 ref:
 branch: main
 commit:
 author:
 email: fluxbot@example.com
 name: FluxBot

Commit and check that the automation object works

Commit the manifeat file and push:

$ git add ./$AUTO_PATH/my-app-auto.yaml
$ git commit -s -m "Add image update automation"
$ git push
# ...

Then sync and check the object status:

$ flux reconcile kustomization --with-source flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ GitRepository reconciliation completed
✔ fetched revision main@sha1:401dd3b550f82581c7d12bb79ade389089c6422f
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ Kustomization reconciliation completed
✔ reconciled revision main@sha1:401dd3b550f82581c7d12bb79ade389089c6422f
$ flux get image update
NAME LAST RUN SUSPENDED READY MESSAGE
my-app-auto 2021-02-08T14:53:43Z False True repository up-to-date

Read on to the next section to see how to change each manifest file to work with Flux v2.

Migrating each manifest to Flux v2

In Flux v1, the annotation

fluxcd.io/automated: "true"

switches automation on for a manifest (a description of a Kubernetes object). For each manifest that has that annotation, you will need to create custom resources to scan for the latest image, and to replace the annotations with field markers.

The following sections explain these steps, using this example Deployment manifest which is initially annotated to work with Flux v1:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-app
 namespace: default
 annotations:
 fluxcd.io/automated: "true"
 fluxcd.io/tag.app: semver:^5.0
spec:
 template:
 spec:
 containers:
 - name: app
 image: ghcr.io/stefanprodan/podinfo:5.0.0

A YAML file may have more than one manifest in it, separated with ---. Be careful to account for each manifest in a file.

You may wish to try migrating the automation of just one file or manifest and follow it through to the end of the guide, before returning here to do the remainder.

How to migrate annotations to image policies

For each image repository that is the subject of automation you will need to create an ImageRepository object, so that the image repository is scanned for tags. The image repository in the example deployment is ghcr.io/stefanprodan/podinfo, which is the image reference minus its tag:

$ cat $CLUSTER_PATH/app/my-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-app
 namespace: default
 annotations:
 fluxcd.io/automated: "true"
 fluxcd.io/tag.app: semver:^5.0
spec:
 template:
 spec:
 containers:
 - name: app
 image: ghcr.io/stefanprodan/podinfo:5.0.0 # <-- image reference

The command-line tool flux will help create a manifest for you. Note that the output is redirected to a file under $AUTO_PATH, so it can be added to the Git repository and synced to the cluster.

$ # the environment variable $AUTO_PATH was set earlier
$ flux create image repository podinfo-image \
 --image ghcr.io/stefanprodan/podinfo \
 --interval 5m \
 --export > ./$AUTO_PATH/podinfo-image.yaml
$ cat ./$AUTO_PATH/podinfo-image.yaml
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: podinfo-image
 namespace: flux-system
spec:
 image: ghcr.io/stefanprodan/podinfo
 interval: 5m0s

Note

If you are using the same image repository in several manifests, you only need one ImageRepository object for it.

Using image registry credentials for scanning

When your image repositories are private, you supply Kubernetes with “image pull secrets” with credentials for accessing the image registry (e.g., DockerHub). The image reflector controller needs the same kind of credentials to scan image repositories.

There are several ways that image pull secrets can be made available for the image reflector controller. The image update tutorial describes how to create or arrange secrets for scanning to use. Also see later in the tutorial for instructions specific to some cloud platforms.

Committing and checking the ImageRepository

Add the ImageRepository manifest to the Git index and commit it:

$ git add ./$AUTO_PATH/podinfo-image.yaml
$ git commit -s -m "Add image repository object for podinfo"
$ git push
# ...

Now you can sync the new commit, and check that the object is working:

$ flux reconcile kustomization --with-source flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ GitRepository reconciliation completed
✔ fetched revision main@sha1:fd2fe8a61d4537bcfa349e4d1dbc480ea699ba8a
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ Kustomization reconciliation completed
✔ reconciled revision main@sha1:fd2fe8a61d4537bcfa349e4d1dbc480ea699ba8a
$ flux get image repository podinfo-image
NAME READY MESSAGE LAST SCAN SUSPENDED
podinfo-image True successful scan, found 16 tags 2021-02-08T14:31:38Z False

Replacing automation annotations

For each field that’s being updated by automation, you’ll need an ImagePolicy object to describe how to select an image for the field value. In the example, the field .image in the container named "app" is the field being updated.

In Flux v1, annotations describe how to select the image to update to, using a prefix. In the example, the prefix is semver::

 annotations:
 fluxcd.io/automated: "true"
 fluxcd.io/tag.app: semver:^5.0

These are the prefixes supported in Flux v1, and what to use in Flux v2:

Flux v1 prefix	Meaning	Flux v2 equivalent
`glob:`	Filter for tags matching the glob pattern, then select the newest by build time	Use sortable tags
`regex:`	Filter for tags matching the regular expression, then select the newest by build time	Use sortable tags
`semver:`	Filter for tags that represent versions, and select the highest version in the given range	Use semver ordering

How to use sortable image tags

To give image tags a useful ordering, you can use a timestamp or serial number as part of each image’s tag, then sort either alphabetically or numerically.

This is a change from Flux v1, in which the build time was fetched from each image’s config, and didn’t need to be included in the image tag. Therefore, this is likely to require a change to your build process.

The guide How to make sortable image tags explains how to change your build process to tag images with a timestamp. This will mean Flux v2 can sort the tags to find the most recently built image.

Filtering the tags in an `ImagePolicy`

The recommended format for image tags using a timestamp is:

<branch>-<sha1>-<timestamp>

The timestamp (or serial number) is the part of the tag that you want to order on. The SHA1 is there so you can trace an image back to the commit from which it was built. You don’t need the branch for sorting, but you may want to include only builds from a specific branch.

Say you want to filter for only images that are from main branch, and pick the most recent. Your ImagePolicy would look like this:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: my-app-policy
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: podinfo-image
 filterTags:
 pattern: '^main-[a-f0-9]+-(?P<ts>[0-9]+)'
 extract: '$ts'
 policy:
 numerical:
 order: asc

The .spec.filterTags.pattern field gives a regular expression that a tag must match to be included. The .spec.filterTags.extract field gives a replacement pattern that can refer back to capture groups in the filter pattern. The extracted values are sorted to find the selected image tag. In this case, the timestamp part of the tag will be extracted and sorted numerically in ascending order. See the reference docs for more examples.

Once you have made sure you have image tags and an ImagePolicy, jump ahead to Checking the ImagePolicy works.

How to use SemVer image tags

The other kind of sorting is by SemVer, picking the highest version from among those included by the filter. A semver range will also filter for tags that fit in the range. For example,

 semver:
 range: ^5.0

includes only tags that have a major version of 5, and selects whichever is the highest.

This can be combined with a regular expression pattern, to filter on other parts of the tags. For example, you might put a target environment as well as the version in your image tags, like dev-v1.0.3.

Then you would use an ImagePolicy similar to this one:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: my-app-policy
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: podinfo-image
 filterTags:
 pattern: '^dev-v(?P<version>.*)'
 extract: '$version'
 policy:
 semver:
 range: '^1.0'

Continue on to the next sections to see an example, and how to check that your ImagePolicy works.

An `ImagePolicy` for the example

The example Deployment has annotations using semver: as a prefix, so the policy object also uses semver:

$ # the environment variable $AUTO_PATH was set earlier
$ flux create image policy my-app-policy \
 --image-ref podinfo-image \
 --semver '^5.0' \
 --export > ./$AUTO_PATH/my-app-policy.yaml
$ cat ./$AUTO_PATH/my-app-policy.yaml
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: my-app-policy
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: podinfo-image
 policy:
 semver:
 range: ^5.0

Checking that the `ImagePolicy` works

Commit the manifest file, and push:

$ git add ./$AUTO_PATH/my-app-policy.yaml
$ git commit -s -m "Add image policy for my-app"
$ git push
# ...

Then you can reconcile and check that the image policy works:

$ flux reconcile kustomization --with-source flux-system
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ GitRepository reconciliation completed
✔ fetched revision main@sha1:7dcf50222499be8c97e22cd37e26bbcda8f70b95
► annotating Kustomization flux-system in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ Kustomization reconciliation completed
✔ reconciled revision main@sha1:7dcf50222499be8c97e22cd37e26bbcda8f70b95
$ flux get image policy flux-system
NAME READY MESSAGE LATEST IMAGE
my-app-policy True Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to: 5.1.4 ghcr.io/stefanprodan/podinfo:5.1.4

How to mark up files for update

The last thing to do in each manifest is to mark the fields that you want to be updated.

In Flux v1, the annotations in a manifest determines the fields to be updated. In the example, the annotations target the image used by the container app:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-app
 namespace: default
 annotations:
 fluxcd.io/automated: "true"
 fluxcd.io/tag.app: semver:^5.0 # <-- `.app` here
spec:
 template:
 spec:
 containers:
 - name: app  # <-- targets `app` here
 image: ghcr.io/stefanprodan/podinfo:5.0.0

This works straight-forwardly for Deployment manifests, but when it comes to HelmRelease manifests, doesn’t work at all for many kinds of resources.

For Flux v2, you mark the field you want to be updated directly, with the namespaced name of the image policy to apply. This is the example Deployment, marked up for Flux v2:

apiVersion: apps/v1
kind: Deployment
metadata:
 namespace: default
 name: my-app
spec:
 template:
 spec:
 containers:
 - name: app
 image: ghcr.io/stefanprodan/podinfo:5.0.0 # {"$imagepolicy": "flux-system:my-app-policy"}

The value flux-system:my-app-policy names the policy that selects the desired image.

This works in the same way for DaemonSet and CronJob manifests. For HelmRelease manifests, put the marker alongside the part of the values that has the image tag. If the image tag is a separate field, you can put :tag on the end of the name, to replace the value with just the selected image’s tag. The image automation guide has examples for HelmRelease and other custom resources.

Committing the marker change and checking that automation works

Referring to the image policy created earlier, you can see the example Deployment does not use the most recent image. When you commit the manifest file with the update marker added, you would expect automation to update the file.

Commit the change that adds an update marker:

$ git add app/my-app.yaml # the filename of the example
$ git commit -s -m "Add update marker to my-app manifest"
$ git push
# ...

Now to check that the automation makes a change:

$ flux reconcile image update my-app-auto
► annotating ImageUpdateAutomation my-app-auto in flux-system namespace
✔ ImageUpdateAutomation annotated
◎ waiting for ImageUpdateAutomation reconciliation
✔ ImageUpdateAutomation reconciliation completed
✔ committed and pushed a92a4b654f520c00cb6c46b2d5e4fb4861aa58fc

Troubleshooting

If a change was not pushed by the image automation, there’s several things you can check:

it’s possible it made a change that is not reported in the latest status – pull from the origin and check the commit log
check that the name used in the marker corresponds to the namespace and name of an ImagePolicy
check that the ImageUpdateAutomation is in the same namespace as the ImagePolicy objects named in markers
check that the image policy and the image repository are both reported as Ready
check that the credentials referenced by the GitRepository object have write permission, and create new credentials if necessary.

As a fallback, you can scan the logs of the automation controller to see if it logged errors:

$ kubectl logs -n flux-system deploy/image-automation-controller

Once you are satisfied that it is working, you can migrate the rest of the manifests using the steps from “Migrating each manifest to Flux v2” above.

Flux: Migrate to the Helm Controller

Mon, 01 Jan 0001 00:00:00 +0000

This guide will learn you everything you need to know to be able to migrate from the Helm Operator to the Helm Controller.

Overview of changes

Support for Helm v2 dropped

The Helm Operator offered support for both Helm v2 and v3, due to Kubernetes client incompatibility issues between the versions. This has blocked the Helm Operator from being able to upgrade to a newer v3 version since the release of 3.2.0.

In combination with the fact that Helm v2 reaches end of life after November 13, 2020, support for Helm v2 has been dropped.

Helm and Git repositories, and even Helm charts are now Custom Resources

When working with the Helm Operator, you had to mount various files to either make it recognize new (private) Helm repositories or make it gain access to Helm and/or Git repositories. While this approach was declarative, it did not provide a great user experience and was at times hard to set up.

By moving this configuration to HelmRepository, GitRepository, Bucket and HelmChart Custom Resources, they can now be declaratively described (including their credentials using references to Secret resources), and applied to the cluster.

The reconciliation of these resources has been offloaded to a dedicated Source Controller, specialized in the acquisition of artifacts from external sources.

The result of this all is an easier and more flexible configuration, with much better observability. Failures are traceable to the level of the resource that lead to a failure, and are easier to resolve. As polling intervals can now be configured per resource, you can customize your repository and/or chart configuration to a much finer grain.

From a technical perspective, this also means less overhead, as the resources managed by the Source Controller can be shared between multiple HelmRelease resources, or even reused by other controllers like the Kustomize Controller.

The `HelmRelease` Custom Resource group domain changed

Due to the Helm Controller becoming part of the extensive set of controller components Flux now has, the Custom Resource group domain has changed from helm.fluxcd.io to helm.toolkit.fluxcd.io.

Together with the new API version (v2 at time of writing), the full apiVersion you use in your YAML document becomes helm.toolkit.fluxcd.io/v2.

The API specification changed (quite a lot), for the better

While developing the Helm Controller, we were given the chance to rethink what a declarative API for driving automated Helm releases would look like. This has, in short, resulted in the following changes:

Extensive configuration options per Helm action (install, upgrade, test, rollback); this includes things like timeouts, disabling hooks, and ignoring failures for tests.
Strategy-based remediation on failures. This makes it possible, for example, to uninstall a release instead of rolling it back after a failed upgrade. The number of retries or keeping the last failed state when the retries are exhausted is now a configurable option.
Better observability. The Status field in the HelmRelease provides a much better view of the current state of the release, including dedicated Ready, Released, TestSuccess, and Remediated conditions.

For a comprehensive overview, see the API spec changes.

Helm storage drift detection no longer relies on dry-runs

The Helm Controller no longer uses dry-runs as a way to detect mutations to the Helm storage. Instead, it uses a simpler model of bookkeeping based on the observed state and revisions. This has resulted in much better performance, a lower memory and CPU footprint, and more reliable drift detection.

No longer supports Helm downloader plugins

We have reduced our usage of Helm packages to a bare minimum (that being: as much as we need to be able to work with chart repositories and charts), and are avoiding shell outs as much as we can.

Given the latter, and the fact that Helm (downloader) plugins work based on shelling out to another command and/or binary, support for this had to be dropped.

We are aware some of our users are using this functionality to be able to retrieve charts from S3 or GCS. The Source Controller already has support for S3 storage compatible buckets ( this includes GCS), and we hope to extend this support in the foreseeable future to be on par with the plugins that offered support for these Helm repository types.

Values from `ConfigMap` and `Secret` resources in other namespaces are no longer supported

Support for values references to ConfigMap and Secret resources in other namespaces than the namespace of the HelmRelease has been dropped, as this allowed information from other namespaces to leak into the composed values for the Helm release.

Values from external source references (URLs) are no longer supported

We initially introduced this feature to support alternative (production focused) values.yaml files that sometimes come with charts. It was also used by users to use generic and/or dynamic values.yaml files in their HelmRelease resources.

The former can now be achieved by defining a ValuesFiles overwrite in the HelmChartTemplateSpec, which will make the Source Controller look for the referenced file in the chart, and overwrite the default values with the contents from that file.

Support for the latter use has been dropped, as it goes against the principles of GitOps and declarative configuration. You can not reliably restore the cluster state from a Git repository if the configuration of a service relies on some URL being available.

You can now merge single values at a given path

There was a long outstanding request for the Helm Operator to support merging single values at a given path.

With the Helm Controller this now possible by defining a targetPath in the ValuesReference, which supports the same formatting as you would supply as an argument to the helm binary using --set [path]=[value]. In addition to this, the referred value can contain the same value formats (e.g. {a,b,c} for a list). You can read more about the available formats and limitations in the Helm documentation.

Support added for depends-on relationships

We have added support for depends-on relationships to install HelmRelease resources in a given order; for example, because a chart relies on the presence of a Custom Resource Definition installed by another HelmRelease resource.

Entries defined in the spec.dependsOn list of the HelmRelease must be in a Ready state before the Helm Controller proceeds with installation and/or upgrade actions.

Note that this does not account for upgrade ordering. Kubernetes only allows applying one resource (HelmRelease in this case) at a time, so there is no way for the controller to know when a dependency HelmRelease may be updated.

Also, circular dependencies between HelmRelease resources must be avoided, otherwise the interdependent HelmRelease resources will never be reconciled.

You can now suspend a HelmRelease

There is a new spec.suspend field, that if set to true causes the Helm Controller to skip reconciliation for the resource. This can be utilized to e.g. temporarily ignore chart changes, and prevent a Helm release from getting upgraded.

Helm releases can target another cluster

We have added support for making Helm releases to other clusters. If the spec.kubeConfig field in the HelmRelease is set, Helm actions will run against the default cluster specified in that KubeConfig instead of the local cluster that is responsible for the reconciliation of the HelmRelease.

The Helm storage is stored on the remote cluster in a namespace that equals to the namespace of the HelmRelease, or the configured spec.storageNamespace. The release itself is made in a namespace that equals to the namespace of the HelmRelease, or the configured spec.targetNamespace. The namespaces are expected to exist, and can for example be created using the Kustomize Controller which has the same cross-cluster support. Other references to Kubernetes resources in the HelmRelease, like ValuesReference resources, are expected to exist on the reconciling cluster.

Added support for notifications and webhooks

Sending notifications and/or alerts to Slack, Microsoft Teams, Discord, or Rocker is now possible using the Notification Controller, Provider Custom Resources and Alert Custom Resources.

It does not stop there, using Receiver Custom Resources you can trigger push based reconciliations from Harbor, GitHub, GitLab, BitBucket or your CI system by making use of the webhook endpoint the resource creates.

Introduction of the `flux` CLI to create and/or generate Custom Resources

With the new flux CLI it is now possible to create and/or generate the Custom Resources mentioned earlier. To generate the YAML for a HelmRepository and HelmRelease resource, you can for example run:

$ flux create source helm podinfo \
 --url=https://stefanprodan.github.io/podinfo \
 --interval=10m \
 --export
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: podinfo
 namespace: flux-system
spec:
 interval: 10m0s
 url: https://stefanprodan.github.io/podinfo

$ flux create helmrelease podinfo \
 --interval=10m \
 --source=HelmRepository/podinfo \
 --chart=podinfo \
 --chart-version=">4.0.0" \
 --export
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: flux-system
spec:
 chart:
 spec:
 chart: podinfo
 sourceRef:
 kind: HelmRepository
 name: podinfo
 version: '>4.0.0'
 interval: 10m0s

API spec changes

The following is an overview of changes to the API spec, including behavioral changes compared to how the Helm Operator performs actions. For a full overview of the new API spec, consult the API spec documentation.

Defining the Helm chart

Helm repository

For the Helm Operator, you used to configure a chart from a Helm repository as follows:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 chart:
 # The repository URL
 repository: https://charts.example.com
 # The name of the chart (without an alias)
 name: my-chart
 # The SemVer version of the chart
 version: 1.2.3

With the Helm Controller, you now create a HelmRepository resource in addition to the HelmRelease you would normally create (for all available fields, consult the Source API reference):

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: my-repository
 namespace: default
spec:
 # The interval at wich to check the upstream for updates
 interval: 10m
 # The repository URL, a valid URL contains at least a protocol and host
 url: https://chart.example.com

If you make use of a private Helm repository, instead of configuring the credentials by mounting a repositories.yaml file, you can now configure the HTTP/S basic auth and/or TLS credentials by referring to a Secret in the same namespace as the HelmRepository:

---
apiVersion: v1
kind: Secret
metadata:
 name: my-repository-creds
 namespace: default
data:
 # HTTP/S basic auth credentials
 username: <base64 encoded username>
 password: <base64 encoded password>
 # TLS credentials (certFile and keyFile, and/or caFile)
 certFile: <base64 encoded certificate>
 keyFile: <base64 encoded key>
 caFile: <base64 encoded CA certificate>
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: my-repository
 namespace: default
spec:
 # ...omitted for brevity
 secretRef:
 name: my-repository-creds

In the HelmRelease, you then use a reference to the HelmRepository resource in the spec.chart.spec (for all available fields, consult the Helm API reference):

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # The interval at which to reconcile the Helm release
 interval: 10m
 chart:
 spec:
 # The name of the chart as made available by the HelmRepository
 # (without any aliases)
 chart: my-chart
 # A fixed SemVer, or any SemVer range
 # (i.e. >=4.0.0 <5.0.0)
 version: 1.2.3
 # The reference to the HelmRepository
 sourceRef:
 kind: HelmRepository
 name: my-repository
 # Optional, defaults to the namespace of the HelmRelease
 namespace: default

The spec.chart.spec values are used by the Helm Controller as a template to create a new HelmChart resource in the same namespace as the sourceRef, to be reconciled by the Source Controller. The Helm Controller watches HelmChart resources for (revision) changes, and performs an installation or upgrade when it notices a change.

Git repository

For the Helm Operator, you used to configure a chart from a Git repository as follows:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 chart:
 # The URL of the Git repository
 git: https://example.com/org/repo
 # The Git branch (or other Git reference)
 ref: master
 # The path of the chart relative to the repository root
 path: ./charts/my-chart

With the Helm Controller, you create a GitRepository resource in addition to the HelmRelease you would normally create (for all available fields, consult the Source API reference:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: my-repository
 namespace: default
spec:
 # The interval at which to check the upstream for updates
 interval: 10m
 # The repository URL, can be a HTTP/S or SSH address
 url: https://example.com/org/repo
 # The Git reference to checkout and monitor for changes
 # (defaults to master)
 # For all available options, see:
 # https://fluxcd.io/flux/components/source/api/v1#source.toolkit.fluxcd.io/v1.GitRepositoryRef
 ref:
 branch: master

If you make use of a private Git repository, instead of configuring the credentials by mounting a private key and making changes to the known_hosts file, you can now configure the credentials for both HTTP/S and SSH by referring to a Secret in the same namespace as the GitRepository:

---
apiVersion: v1
kind: Secret
metadata:
 name: my-repository-creds
 namespace: default
data:
 # HTTP/S basic auth credentials
 username: <base64 encoded username>
 password: <base64 encoded password>
 # SSH credentials
 identity: <base64 encoded private key>
 identity.pub: <base64 public key>
 known_hosts: <base64 encoded known_hosts>
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: my-repository
 namespace: default
spec:
 # ...omitted for brevity
 secretRef:
 name: my-repository-creds

In the HelmRelease, you then use a reference to the GitRepository resource in the spec.chart.spec (for all available fields, consult the Helm API reference):

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # The interval at which to reconcile the Helm release
 interval: 10m
 chart:
 spec:
 # The path of the chart relative to the repository root
 chart: ./charts/my-chart
 # The reference to the GitRepository
 sourceRef:
 kind: GitRepository
 name: my-repository
 # Optional, defaults to the namespace of the HelmRelease
 namespace: default

Defining values

Inlined values

Inlined values (defined in the spec.values of the HelmRelease) still work as with the Helm operator. It represents a YAML map as you would put in a file and supply to helm with -f values.yaml, but inlined into the HelmRelease manifest:

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 values:
 foo: value1
 bar:
 baz: value2
 oof:
 - item1
 - item2

Values from sources

As described in the overview of changes, there have been multiple changes to the way you can refer to values from sources (like ConfigMap and Secret references), including the drop of support for external source (URL) references and added support for merging single values at a specific path.

Values are still merged in the order given, with later values overwriting earlier. The values from sources always have a lower priority than the values inlined in the HelmRelease via the spec.values key.

`ConfigMap` and `Secret` references

ConfigMap and Secret references used to be defined as follows:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 valuesFrom:
 - configMapKeyRef:
 name: my-config-values
 namespace: my-ns
 key: values.yaml
 optional: false
 - secretKeyRef:
 name: my-secret-values
 namespace: my-ns
 key: values.yaml
 optional: true

In the new API spec the individual configMapKeyRef and secretKeyRef objects are bundled into a single ValuesReference which does no longer allow refering to resources in other namespaces:

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 valuesFrom:
 - kind: ConfigMap
 name: my-config-values
 valuesKey: values.yaml
 optional: false
 - kind: Secret
 name: my-secret-values
 valuesKey: values.yaml
 optional: true

Another thing to take note of is that the behavior for values references marked as optional has changed. When set, a “not found” error for the values reference is ignored, but any valuesKey, targetPath or transient error will still result in a reconciliation failure.

Chart file references

With the Helm Operator it was possible to refer to an alternative values file (for e.g. production usage) in the directory of a chart from a Git repository:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 valuesFrom:
 # Values file to merge in,
 # expected to be a relative path in the chart directory
 - chartFileRef:
 path: values-prod.yaml

With the Helm Controller, this declaration has moved to the spec.chart.spec, and the feature is no longer limited to charts from a Git repository:

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 chart:
 spec:
 chart: my-chart
 version: 1.2.3
 # Alternative values file to use as the default values,
 # expected to be a relative path in the sourceRef
 valuesFiles:
 - values.yaml
 - values-prod.yaml
 sourceRef:
 kind: HelmRepository
 name: my-repository

When valuesFiles is defined, the chart will be (re)packaged with the values from the referenced files as the default values, merged in the order they appear. Note that this behavior is different from the Helm Operator as the default values (values.yaml) are not merged by default and must be explicitly added to the list.

External source references

While the support for external source references has been dropped, it is possible to work around this limitation by creating a CronJob that periodically fetches the values from an external URL and saves them to a ConfigMap or Secret resource.

First, create a ServiceAccount, Role and RoleBinding capable of updating a limited set of ConfigMap resources:

---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: values-fetcher
 namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: configmap-updater
 namespace: default
rules:
- apiGroups: [""]
 resources: ["configmaps"]
 # ResourceNames limits the access of the role to
 # a defined set of ConfigMap resources
 resourceNames: ["my-external-values"]
 verbs: ["patch", "get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: update-values-configmaps
 namespace: default
subjects:
- kind: ServiceAccount
 name: values-fetcher
 namespace: default
roleRef:
 kind: Role
 name: configmap-updater
 apiGroup: rbac.authorization.k8s.io

As resourceNames scoping in the Role does not allow restricting create requests, we need to create empty placeholder(s) for the ConfigMap resource(s) that will hold the fetched values:

---
apiVersion: v1
kind: ConfigMap
metadata:
 name: my-external-values
 namespace: default
data: {}

Lastly, create a CronJob that uses the ServiceAccount defined above, fetches the external values on an interval, and applies them to the ConfigMap:

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
 name: fetch-external-values
spec:
 concurrencyPolicy: Forbid
 schedule: "*/5 * * * *"
 successfulJobsHistoryLimit: 3
 failedJobsHistoryLimit: 3
 jobTemplate:
 spec:
 template:
 spec:
 serviceAccountName: values-fetcher
 containers:
 - name: kubectl
 image: bitnami/kubectl:1.19
 volumeMounts:
 - mountPath: /tmp
 name: tmp-volume
 command:
 - sh
 - -c
 args:
 - >-
 curl -f -# https://example.com/path/to/values.yaml -o /tmp/values.yaml &&
 kubectl create configmap my-external-values --from-file=/tmp/values.yaml -oyaml --dry-run=client |
 kubectl apply -f -
 volumes:
 - name: tmp-volume
 emptyDir:
 medium: Memory
 restartPolicy: OnFailure

You can now refer to the my-external-values ConfigMap resource in your HelmRelease:

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 valuesFrom:
 - kind: ConfigMap
 name: my-external-values

Defining release options

With the Helm Operator the release options used to be configured in the spec of the HelmRelease and applied to both Helm install and upgrade actions.

This has changed for the Helm Controller, where some defaults can be defined in the spec, but specific action configurations and overwrites for the defaults can be defined in the spec.install, spec.upgrade and spec.test sections of the HelmRelease.

Defining a rollback / uninstall configuration

With the Helm Operator, uninstalling a release after an installation failure was done automatically, and rolling back from a faulty upgrade and configuring options like retries was done as follows:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 rollback:
 enable: true
 retries: true
 maxRetries: 5
 disableHooks: false
 force: false
 recreate: false
 timeout: 300

The Helm Controller offers an extensive set of configuration options to remediate when a Helm release fails, using spec.install.remediation, spec.upgrade.remediation, spec.rollback and spec.uninstall. Some of the new features include the option to remediate with an uninstall after an upgrade failure, and the option to keep a failed release for debugging purposes when it has run out of retries.

Automated uninstalls

The configuration below mimics the uninstall behavior of the Helm Operator (for all available fields, consult the InstallRemediation and Uninstall API references):

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 install:
 # Remediation configuration for when the Helm install
 # (or sequent Helm test) action fails
 remediation:
 # Number of retries that should be attempted on failures before
 # bailing, a negative integer equals to unlimited retries
 retries: -1
 # Configuration options for the Helm uninstall action
 uninstall:
 timeout: 5m
 disableHooks: false
 keepHistory: false

Automated rollbacks

The configuration below shows an automated rollback configuration that equals the configuration for the Helm Operator showed above (for all available fields, consult the UpgradeRemediation and Rollback API references):

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 upgrade:
 # Remediaton configuration for when an Helm upgrade action fails
 remediation:
 # Amount of retries to attempt after a failure,
 # setting this to 0 means no remedation will be
 # attempted
 retries: 5
 # Configuration options for the Helm rollback action
 rollback:
 timeout: 5m
 disableWait: false
 disableHooks: false
 recreate: false
 force: false
 cleanupOnFail: false

Migration strategy

Due to the high number of changes to the API spec, there are no detailed instructions available to provide a simple migration path. But there is a simple procedure to follow, which combined with the detailed list of API spec changes should make the migration path relatively easy.

Here are some things to know:

The Helm Controller will ignore the old custom resources (and the Helm Operator will ignore the new resources).
Deleting a resource while the corresponding controller is running will result in the Helm release also being deleted.
Deleting a CustomResourceDefinition will also delete all custom resources of that kind.
If both the Helm Controller and Helm Operator are running, and both a new and old custom resources define a release, they will fight over the release.
The Helm Controller will always perform an upgrade the first time it encounters a new HelmRelease for an existing release; this is due to the changes to release mechanics and bookkeeping.

The safest way to upgrade is to avoid deletions and fights by stopping the Helm Operator. Once the operator is not running, it is safe to deploy the Helm Controller (e.g., by following the Get Started guide, utilizing flux install, or using the manifests from the release page), and start replacing the old resources with new resources. You can keep the old resources around during this process, since the Helm Controller will ignore them.

Steps

The recommended migration steps for a single HelmRelease are as follows:

Ensure the Helm Operator is not running, as otherwise the Helm Controller and Helm Operator will fight over the release.
Create a GitRepository or HelmRepository resource for the HelmRelease, including any Secret that may be required to access the source. Note that it is possible for multiple HelmRelease resources to share a GitRepository or HelmRepository resource.
Create a new HelmRelease resource ( with the helm.toolkit.fluxcd.io group domain), define the spec.releaseName (plus the spec.targetNamespace and spec.storageNamespace if applicable) to match that of the existing release, and rewrite the configuration to adhere to the API spec changes.
Confirm the Helm Controller successfully upgrades the release.

Example

As a full example, this is an old resource:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
 name: podinfo
 namespace: default
spec:
 chart:
 repository: https://stefanprodan.github.io/podinfo
 name: podinfo
 version: 5.0.3
 values:
 replicaCount: 1

The custom resources for the Helm Controller would be:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: podinfo
 namespace: default
spec:
 interval: 10m
 url: https://stefanprodan.github.io/podinfo
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: default
spec:
 interval: 5m
 releaseName: default-podinfo
 chart:
 spec:
 chart: podinfo
 version: 5.0.3
 sourceRef:
 kind: HelmRepository
 name: podinfo
 interval: 10m
 values:
 replicaCount: 1

Migrating gradually

Gradually migrating to the Helm Controller is possible by scaling down the Helm Operator while you move over resources, and scaling it up again once you have migrated some of the releases to the Helm Controller.

While doing this, make sure that once you scale up the Helm Operator again, there are no old and new HelmRelease resources pointing towards the same release, as they will fight over the release.

Alternatively, you can gradually migrate per namespace without ever needing to shut the Helm Operator down, enabling no continuous delivery interruption on most namespaces. To do so, you can customize the Helm Operator roles associated to its ServiceAccount to prevent it from interfering with the Helm Controller in namespaces you are migrating. First, create a new ClusterRole for the Helm Operator to operate in “read-only” mode cluster-wide:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: helm-operator-ro
rules:
 - apiGroups: ['*']
 resources: ['*']
 verbs:
 - get
 - watch
 - list
 - nonResourceURLs: ['*']
 verbs: ['*']

By default, the helm-operator ServiceAccount is bound to a ClusterRole that allows it to create, patch and delete resources in all namespaces. Bind the ServiceAccount to the new helm-operator-ro ClusterRole:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: helm-operator
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
- name: helm-operator
+ name: helm-operator-ro
subjects:
 - kind: ServiceAccount
 name: helm-operator
 namespace: flux

Finally, create RoleBindings for each namespace, but the one you are currently migrating:

# Create a `RoleBinding` for each namespace the Helm Operator is allowed to process `HelmReleases` in
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: helm-operator
 namespace: helm-operator-watched-namespace
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: helm-operator
subjects:
 - name: helm-operator
 namespace: flux
 kind: ServiceAccount
# Do not create the following to prevent the Helm Operator from watching `HelmReleases` in `helm-controller-watched-namespace`
# ---
# apiVersion: rbac.authorization.k8s.io/v1
# kind: RoleBinding
# metadata:
# name: helm-operator
# namespace: helm-controller-watched-namespace
# roleRef:
# apiGroup: rbac.authorization.k8s.io
# kind: ClusterRole
# name: helm-operator
# subjects:
# - name: helm-operator
# namespace: flux
# kind: ServiceAccount

If you are using the Helm Operator chart, make sure to set rbac.create to false in order to take over ClusterRoleBindings and RoleBindings as you wish.

Deleting old resources

Once you have migrated all your HelmRelease resources to the Helm Controller. You can remove all of the old resources by removing the old Custom Resource Definition.

kubectl delete crd helmreleases.helm.fluxcd.io

Frequently Asked Questions

Are automated image updates supported?

Yes, image updates are supported for HelmRelease as well as any other Kubernetes custom resources stored in Git. See the Image Update Guide for more information.

How do I automatically apply my `HelmRelease` resources to the cluster?

If you are currently a Flux v1 user, you can commit the HelmRelease resources to Git, and Flux will automatically apply them to the cluster like any other resource.

If you are not a Flux v1 user or want to fully migrate to Flux v2, the Kustomize Controller will serve your needs.

I am still running Helm v2, what is the right upgrade path for me?

Migrate your Helm v2 releases to v3 using the Helm Operator’s migration feature, or make use of the helm-2to3 plugin directly, before continuing following the migration steps.

Is the Helm Controller ready for production?

Probably, but with some side notes:

It is still under active development, and while our focus has been to stabilize the API as much as we can during the first development phase, we do not guarantee there will not be any breaking changes before we reach General Availability. We are however committed to provide conversion webhooks for upcoming API versions.
There may be (internal) behavioral changes in upcoming releases, but they should be aimed at further stabilizing the Helm Controller itself, solving edge case issues, providing better logging, observability, and/or other improvements.

Can I use Helm Controller standalone?

Helm Controller depends on Source Controller, you can install both controllers and manager Helm releases in a declarative way without GitOps.

For more details please see this answer.

I have another question

Given the amount of changes, it is quite possible that this document did not provide you with a clear answer for you specific setup. If this applies to you, do not hesitate to ask for help in the GitHub Discussions or on the #flux CNCF Slack channel!

Flux: FAQ

Mon, 01 Jan 0001 00:00:00 +0000

Flux v1 vs v2 questions

What does Flux v2 mean for Flux?

Flux v1 is a monolithic do-it-all operator; Flux v2 separates the functionalities into specialized controllers, collectively called the GitOps Toolkit.

You can install and operate Flux v2 simply using the flux command. You can easily pick and choose the functionality you need and extend it to serve your own purposes.

We went through the following steps for our community:

Put Flux v1 into maintenance mode (no new features being added; bugfixes and CVEs patched only).
Continued work on the Flux v2 roadmap.
We provided transition guides for specific user groups, e.g. users of Flux v1 in read-only mode, or of Helm Operator v1, etc. once the functionality was integrated into Flux v2 and it’s deemed “ready”.
Once the use-cases of Flux v1 were covered, we promised to continue supporting Flux v1 for 6 months.
We finally archived Flux Legacy and Helm Operator.

Why did you rewrite Flux?

Flux v2 implements its functionality in individual controllers, which allowed us to address long-standing feature requests much more easily.

By basing these controllers on modern Kubernetes tooling (controller-runtime libraries), they can be dynamically configured with Kubernetes custom resources either by cluster admins or by other automated tools – and you get greatly increased observability.

This gave us the opportunity to build Flux v2 with the top Flux v1 feature requests in mind:

Supporting multiple source Git repositories
Operational insight through health checks, events and alerts
Multi-tenancy capabilities, like applying each source repository with its own set of permissions

On top of that, testing the individual components and understanding the codebase becomes a lot easier.

What are significant new differences between Flux v1 and Flux v2?

Reconciliation

Flux v1	Flux v2
Limited to a single Git repository	Multiple Git repositories
Declarative config via arguments in the Flux deployment	`GitRepository` custom resource, which produces an artifact which can be reconciled by other controllers
Follow `HEAD` of Git branches	Supports Git branches, pinning on commits and tags, follow SemVer tag ranges
Suspending of reconciliation by downscaling Flux deployment	Reconciliation can be paused per resource by suspending the `GitRepository`
Credentials config via Arguments and/or Secret volume mounts in the Flux pod	Credentials config per `GitRepository` resource: SSH private key, HTTP/S username/password/token, OpenPGP public keys
Ignoring resources with `fluxcd.io/ignore: "true"` annotation	Ignoring resources with `kustomize.toolkit.fluxcd.io/reconcile: disabled` annotation

`kustomize` support

Flux v1	Flux v2
Declarative config through `.flux.yaml` files in the Git repository	Declarative config through a `Kustomization` custom resource, consuming the artifact from the GitRepository
Manifests are generated via shell exec and then reconciled by `fluxd`	Generation, server-side validation, and reconciliation is handled by a specialised `kustomize-controller`
Reconciliation using the service account of the Flux deployment	Support for service account impersonation
Garbage collection needs cluster role binding for Flux to query the Kubernetes discovery API	Garbage collection needs no cluster role binding or access to Kubernetes discovery API
Support for custom commands and generators executed by fluxd in a POSIX shell	No support for custom commands

Helm integration

Flux v1	Flux v2
Declarative config in a single Helm custom resource	Declarative config through `HelmRepository`, `GitRepository`, `Bucket`, `HelmChart` and `HelmRelease` custom resources
Chart synchronisation embedded in the operator	Extensive release configuration options, and a reconciliation interval per source
Support for fixed SemVer versions from Helm repositories	Support for SemVer ranges for `HelmChart` resources
Git repository synchronisation on a global interval	Planned support for charts from GitRepository sources
Limited observability via the status object of the HelmRelease resource	Better observability via the HelmRelease status object, Kubernetes events, and notifications
Resource heavy, relatively slow	Better performance
Chart changes from Git sources are determined from Git metadata	Chart changes must be accompanied by a version bump in `Chart.yaml` to produce a new artifact

Notifications, webhooks, observability

Flux v1	Flux v2
Emits “custom Flux events” to a webhook endpoint	Emits Kubernetes events for included custom resources
RPC endpoint can be configured to a 3rd party solution like FluxCloud to be forwarded as notifications to e.g. Slack	Flux v2 components can be configured to POST the events to a `notification-controller` endpoint. Selective forwarding of POSTed events as notifications using `Provider` and `Alert` custom resources.
Webhook receiver is a side-project	Webhook receiver, handling a wide range of platforms, is included
Unstructured logging	Structured logging for all components
Custom Prometheus metrics	Generic / common `controller-runtime` Prometheus metrics

Are there any breaking changes?

In Flux v1 Kustomize support was implemented through .flux.yaml files in the Git repository. As indicated in the comparison table above, while this approach worked, we found it to be error-prone and hard to debug. The new Kustomization CR should make troubleshooting much easier. Unfortunately we needed to drop the support for custom commands as running arbitrary shell scripts in-cluster poses serious security concerns.
Helm users: we redesigned the HelmRelease API and the automation will work quite differently, so upgrading to HelmRelease v2 will require a little work from you, but you will gain more flexibility, better observability and performance.

In an announcement in August 2019, the expectation was set that the Flux project would integrate the GitOps Engine, then being factored out of ArgoCD. Since the result would be backward-incompatible, it would require a major version bump: Flux v2.

After experimentation and considerable thought, we (the maintainers) have found a path to Flux v2 that we think better serves our vision of GitOps: the GitOps Toolkit. In consequence, we do not now plan to integrate GitOps Engine into Flux.

How can I get involved?

There are a variety of ways and we look forward to having you on board building the future of GitOps together:

Discuss the direction of Flux v2 with us
Join us in #flux-dev on the CNCF Slack
Check out our contributor docs
Take a look at the roadmap for Flux v2

Flux: Timetable

Mon, 01 Jan 0001 00:00:00 +0000

Flux Legacy has reached end of life.

Flux Legacy and Helm Operator have been archived.

We strongly recommend you familiarise yourself with the newest Flux and migrate as soon as possible.

Date	Flux 1	Flux 2 CLI	GOTK¹
Oct 6, 2020	Maintenance Mode Flux 1 releases only include critical bug fixes (which don’t require changing Flux 1 architecture), and security patches (for OS packages, Go runtime, and kubectl). No new features Existing projects encouraged to test migration to Flux 2 pre-releases in non-production	Development Mode Working to finish parity with Flux 1 New projects encouraged to test Flux 2 pre-releases in non-production	All Alpha²
Feb 18, 2021	Partial Migration Mode Existing projects encouraged to migrate to `v1beta1`/`v2beta1` if you only use those features (Flux 1 read-only mode, and Helm Operator) Existing projects encouraged to test image automation Alpha in non-production	Feature Parity	Image Automation Alpha. All others reached Feature Parity, Beta
June 30, 2021	Superseded All existing projects encouraged to migrate to Flux 2, and report any bugs Flux 1 Helm Operator code freeze – no further updates except CVEs	Needs further testing, may get breaking changes CLI needs further user testing during this migration period	All Beta, Production Ready³ All Flux 1 features stable and supported in Flux 2 Promoting Alpha versions to Beta makes this Production Ready
May 1, 2022	Migration and security support only Flux 1 releases only include security patches (no bug fixes) Maintainers support users with migration to Flux 2 only, no longer with Flux 1 issues	Beta, Production Ready	All Beta, Production Ready
November 1, 2022	Archived Flux 1 obsolete, no further releases or maintainer support Flux 1 repo archived helm-operator repo archived	Public release candidate, Production Ready CLI commits to backwards compatibility moving forward CLI follows kubectl style backwards compatibility support: +1 -1 MINOR version for server components (e.g., APIs, Controllers, validation webhooks)	All stable, Production Ready

GOTK is shorthand for the GitOps Toolkit APIs and Controllers ↩︎
Versioning: Flux 2 is a multi-service architecture, so requires a more complex explanation than Flux 1:
- Flux 2 CLI follows Semantic Versioning scheme
- The GitOps Toolkit APIs follow the Kubernetes API versioning pattern. See Roadmap for component versions.
- These are coordinated for cross-compatibility: For each Flux 2 CLI tag, CLI and GOTK versions are end-to-end tested together, so you may safely upgrade from one MINOR/PATCH version to another.
↩︎
The GOTK Custom Resource Definitions which are at v1beta1, v1beta2, v2beta1 and v2beta2 and their controllers are considered stable and production ready. Going forward, breaking changes to the beta CRDs will be accompanied by a conversion mechanism. ↩︎

Flux – Migrate from Flux v1

Flux: Migrate from Flux v1 to v2

Prerequisites

Install Flux v2 CLI

GitOps migration

'flux bootstrap' target

In-place migration

Flux read-only mode

Uninstall Flux v1

Flux with Kustomize

Uninstall Flux v1

Flux with Slack notifications

Flux debugging

Flux: Migrating image update automation to Flux v2

Overview of changes between v1 and v2

Automation is now controlled by custom resources

Selecting an image is more flexible

Fields to update are explicitly marked

Preparing for migration

Where to keep ImageRepository, ImagePolicy and ImageUpdateAutomation manifests

Migration on a branch

Installing the command-line tool flux

Running the automation controllers

After flux bootstrap

After migrating Flux v1 in place

After committing a Flux v2 configuration to Git

Controlling automation with an ImageUpdateAutomation object

Replacing the Git credentials secret

Replacing an SSH key secret

Replacing a username/password secret

Checking the new credentials

Making an automation object

Commit and check that the automation object works

Migrating each manifest to Flux v2

How to migrate annotations to image policies

Note

Using image registry credentials for scanning

Committing and checking the ImageRepository

Replacing automation annotations

How to use sortable image tags

Filtering the tags in an ImagePolicy

How to use SemVer image tags

An ImagePolicy for the example

Checking that the ImagePolicy works

How to mark up files for update

Committing the marker change and checking that automation works

Troubleshooting

Flux: Migrate to the Helm Controller

Overview of changes

Support for Helm v2 dropped

Helm and Git repositories, and even Helm charts are now Custom Resources

The HelmRelease Custom Resource group domain changed

The API specification changed (quite a lot), for the better

Helm storage drift detection no longer relies on dry-runs

No longer supports Helm downloader plugins

Values from ConfigMap and Secret resources in other namespaces are no longer supported

Values from external source references (URLs) are no longer supported

You can now merge single values at a given path

Support added for depends-on relationships

You can now suspend a HelmRelease

Helm releases can target another cluster

Added support for notifications and webhooks

Introduction of the flux CLI to create and/or generate Custom Resources

API spec changes

Defining the Helm chart

Helm repository

Git repository

Defining values

Inlined values

Values from sources

ConfigMap and Secret references

Chart file references

External source references

Defining release options

Defining a rollback / uninstall configuration

Automated uninstalls

Automated rollbacks

Migration strategy

Steps

Example

Where to keep `ImageRepository`, `ImagePolicy` and `ImageUpdateAutomation` manifests

Installing the command-line tool `flux`

After `flux bootstrap`

Controlling automation with an `ImageUpdateAutomation` object

Filtering the tags in an `ImagePolicy`

An `ImagePolicy` for the example

Checking that the `ImagePolicy` works

The `HelmRelease` Custom Resource group domain changed

Values from `ConfigMap` and `Secret` resources in other namespaces are no longer supported

Introduction of the `flux` CLI to create and/or generate Custom Resources

`ConfigMap` and `Secret` references

How do I automatically apply my `HelmRelease` resources to the cluster?

`kustomize` support