Flux – Use Cases

Flux: Running pre and post-deployment jobs with Flux

Mon, 01 Jan 0001 00:00:00 +0000

Additional considerations have to be made when managing Kubernetes Jobs with Flux. By default, if you were to have Flux reconcile a Job resource, it would apply it once to the cluster, the Job would create a Pod that can either error or run to completion. Attempting to update the Job manifest after it has been applied to the cluster will not be allowed, as changes to the Job spec.Completions, spec.Selector and spec.Template are not permitted by the Kubernetes API. To be able to update a Kubernetes Job, the Job has to be recreated by first being removed and then reapplied to the cluster.

Repository structure

A typical use case for running Kubernetes Jobs with Flux is to implement pre-deployment tasks for e.g. database scheme migration and post-deployment jobs (like cache refresh).

This requires separate Flux Kustomization resources that depend on each other: one for running the pre-deployment Jobs, one to deploy the application, and a 3rd one for running the post-deployment Jobs.

Example of an application configuration repository:

├── pre-deploy
│   └── migration.job.yaml
├── deploy
│   ├── deployment.yaml
│   ├── ingress.yaml
│   └── service.yaml
├── post-deploy
│   └── cache.job.yaml
└── flux
 ├── pre-deploy.yaml
 ├── deploy.yaml
 └── post-deploy.yaml

Configure the deployment pipeline

Given a Job in the path ./pre-deploy/migration.job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
 name: db-migration
spec:
 template:
 spec:
 restartPolicy: Never
 containers:
 - name: migration
 image: ghcr.io/org/my-app:v1.0.0
 command:
 - sh
 - -c
 - echo "starting db migration"

And a Flux Kustomization that reconciles it at ./flux/pre-deploy.yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: app-pre-deploy
spec:
 sourceRef:
 kind: GitRepository
 name: my-app
 path: "./pre-deploy/"
 interval: 60m
 timeout: 5m
 prune: true
 wait: true
 force: true

Setting spec.force to true will make Flux recreate the Job when any immutable field is changed, forcing the Job to run every time the container image tag changes. Setting spec.wait to true makes Flux wait for the Job to complete before it is considered ready.

To deploy the application after the migration job, we define a Flux Kustomization that depends on the migration one.

Example of ./flux/deploy.yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: app-deploy
spec:
 dependsOn:
 - name: app-pre-deploy
 sourceRef:
 kind: GitRepository
 name: my-app
 path: "./deploy/"
 interval: 60m
 timeout: 5m
 prune: true
 wait: true

This means that the app-deploy Kustomization will wait until all the Jobs in app-pre-deploy run to completion. If the Job fails, the app changes will not be applied by the app-deploy Kustomization.

And finally we can define a Flux Kustomization that depends on app-deploy to run Kubernetes Jobs after the application was upgraded.

Example of ./flux/post-deploy.yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: app-post-deploy
spec:
 dependsOn:
 - name: app-deploy
 sourceRef:
 kind: GitRepository
 name: my-app
 path: "./post-deploy/"
 interval: 60m
 timeout: 5m
 prune: true
 wait: true
 force: true

This configuration works best when the Jobs are using the same image and tag as the application being deployed. When a new version of the application is deployed, the image tags are updated. The update of the image tag will force a recreation of the Jobs. The application will be updated after the pre-deployment Jobs have run successfully, and the post-deployment Jobs will execute only if the app rolling update has completed.

Flux: Karmada + Flux

Mon, 01 Jan 0001 00:00:00 +0000

Disclaimer

Note that this guide is not for doing GitOps, but for managing Helm releases for applications among multiple clusters.

Also note that this guide needs review in consideration of Flux v2.0.0, and likely needs to be refreshed.

Expect this doc to either be archived soon, or to receive some overhaul.

Background

Karmada is a Kubernetes management system that enables you to run your cloud-native applications across multiple Kubernetes clusters and clouds, with no changes to your applications. By speaking Kubernetes-native APIs and providing advanced scheduling capabilities, Karmada enables truly open, multi-cloud Kubernetes. With Karmada’s centralized multi-cloud management, users can easily distribute and manage Helm releases in multiple clusters based on powerful Flux APIs.

Karmada Setup

Steps described in this document have been tested on Karmada 1.0, 1.1 and 1.2. To start up Karmada, you can refer to here. If you just want to try Karmada, we recommend building a development environment by hack/local-up-karmada.sh.

git clone https://github.com/karmada-io/karmada
cd karmada
hack/local-up-karmada.sh

After that, you will start a Kubernetes cluster by kind to run the Karmada control plane and create member clusters managed by Karmada.

kubectl get clusters --kubeconfig ~/.kube/karmada.config

You can use the command above to check registered clusters, and you will get similar output as follows:

NAME VERSION MODE READY AGE
member1 v1.23.4 Push True 7m38s
member2 v1.23.4 Push True 7m35s
member3 v1.23.4 Pull True 7m27s

Flux Installation

In Karmada control plane, you need to install Flux crds but do not need controllers to reconcile them. They are treated as resource templates, not specific resource instances. Based on work API here, they will be encapsulated as a work object deliverd to member clusters and reconciled by Flux controllers in member clusters finally.

kubectl apply -k github.com/fluxcd/flux2/manifests/crds?ref=main --kubeconfig ~/.kube/karmada.config

For testing purposes, we’ll install Flux on member clusters without storing its manifests in a Git repository:

flux install --kubeconfig ~/.kube/members.config --context member1
flux install --kubeconfig ~/.kube/members.config --context member2

Please refer to the documentations here for more ways to set up Flux in details.

Tip

If you want to manage Helm releases across your fleet of clusters, Flux must be installed on each cluster.

Helm release propagation

If you want to propagate Helm releases for your apps to member clusters, you can refer to the guide below.

Define a Flux HelmRepository and a HelmRelease manifest in Karmada Control plane. They will serve as resource templates.

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: podinfo
spec:
 interval: 1m
 url: https://stefanprodan.github.io/podinfo 
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
spec:
 interval: 5m
 chart:
 spec:
 chart: podinfo
 version: 5.0.3
 sourceRef:
 kind: HelmRepository
 name: podinfo

Define a Karmada PropagationPolicy that will propagate them to member clusters:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
 name: helm-repo
spec:
 resourceSelectors:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: HelmRepository
 name: podinfo
 placement:
 clusterAffinity:
 clusterNames:
 - member1
 - member2
---
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
 name: helm-release
spec:
 resourceSelectors:
 - apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 name: podinfo
 placement:
 clusterAffinity:
 clusterNames:
 - member1
 - member2

The above configuration is for propagating the Flux objects to member1 and member2 clusters.

Apply those manifests to the Karmada-apiserver:

kubectl apply -f ../helm/ --kubeconfig ~/.kube/karmada.config

The output is similar to:

helmrelease.helm.toolkit.fluxcd.io/podinfo created
helmrepository.source.toolkit.fluxcd.io/podinfo created
propagationpolicy.policy.karmada.io/helm-release created
propagationpolicy.policy.karmada.io/helm-repo created

Switch to the distributed cluster and verify:

helm --kubeconfig ~/.kube/members.config --kube-context member1 list

The output is similar to:

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
podinfo default 1 2022-05-27 01:44:35.24229175 +0000 UTC deployed podinfo-5.0.3 5.0.3

Based on Karmada’s propagation policy, you can schedule Helm releases to your desired cluster flexibly, just like Kubernetes scheduling Pods to the desired node.

Customize the Helm release for specific clusters

The example above shows how to distribute the same Helm release to multiple clusters in Karmada. Besides, you can use Karmada’s OverridePolicy to customize applications for specific clusters. For example, If you just want to change replicas in member1, you can refer to the overridePolicy below.

Define a Karmada OverridePolicy:

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
 name: example-override
 namespace: default
spec:
 resourceSelectors:
 - apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 name: podinfo
 overrideRules:
 - targetCluster:
 clusterNames:
 - member1
 overriders:
 plaintext:
 - path: "/spec/values"
 operator: add
 value:
 replicaCount: 2

Apply the manifests to the Karmada-apiserver:

kubectl apply -f example-override.yaml --kubeconfig ~/.kube/karmada.config

The output is similar to:

overridepolicy.policy.karmada.io/example-override configured

After applying the above policy in Karmada control plane, you will find that replicas in member1 has changed to 2, but those in member2 keep the same.

kubectl --kubeconfig ~/.kube/members.config --context member1 get po

The output is similar to:

NAME READY STATUS RESTARTS AGE
podinfo-68979685bc-6wz6s 1/1 Running 0 6m28s
podinfo-68979685bc-dz9f6 1/1 Running 0 7m42s

Flux: Flux for Helm Users

Mon, 01 Jan 0001 00:00:00 +0000

Welcome Helm users! We think Flux’s Helm Controller is the best way to do Helm according to GitOps principles, and we’re dedicated to doing what we can to help you feel the same way.

What Does Flux add to Helm?

Helm 3 was designed with both a client and an SDK, but no running software agents. This architecture intended anything outside of the client scope to be addressed by other tools in the ecosystem, which could then make use of Helm’s SDK.

Built on Kubernetes controller-runtime, Flux’s Helm Controller is an example of a mature software agent that uses Helm’s SDK to full effect.

Flux’s biggest addition to Helm is a structured declaration layer for your releases that automatically gets reconciled to your cluster based on your configured rules:

While the Helm client commands let you imperatively do things
Flux Helm Custom Resources let you declare what you want the Helm SDK to do automatically

Additional benefits Flux adds to Helm include:

Managing / structuring multiple environments
A control loop, with configurable retry logic
Automated drift detection between the desired and actual state of your operations
Automated responses to that drift, including reconciliation, notifications, and unified logging

Getting Started

The simplest way to explain is by example. Lets translate imperative Helm commands to Flux Helm Controller Custom Resources:

Helm client:

helm repo add traefik https://helm.traefik.io/traefik
helm install my-traefik traefik/traefik \
 --version 9.18.2 \
 --namespace traefik

Flux client:

flux create source helm traefik --url https://helm.traefik.io/traefik --namespace traefik
flux create helmrelease my-traefik --chart traefik \
 --source HelmRepository/traefik \
 --chart-version 9.18.2 \
 --namespace traefik

The main difference is that the Flux client will not imperatively create resources in the cluster. Instead, these commands create Custom Resource files, which are committed to version control as instructions only (note: you may use the --export flag to manage any file edits with finer grained control before pushing to version control). Separately, the Flux Helm Controller automatically reconciles these instructions with the running state of your cluster based on your configured rules.

Let’s check out what the Custom Resource files look like:

# /flux/boot/traefik/helmrepo.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: traefik
 namespace: traefik
spec:
 interval: 1m0s
 url: https://helm.traefik.io/traefik

# /flux/boot/traefik/helmrelease.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-traefik
 namespace: traefik
spec:
 chart:
 spec:
 chart: traefik
 sourceRef:
 kind: HelmRepository
 name: traefik
 version: 9.18.2
 interval: 1m0s

Once these are applied to your cluster, the Flux Helm Controller automatically uses the Helm SDK to do your bidding according to the rules you’ve set.

Why is this important? If you or your team has ever collaborated with multiple engineers on one or more apps, and/or in more than one namespace or cluster, you probably have a good idea of how declarative, automatic reconciliation can help solve common problems. If not, or either way, you may want to check out this short introduction to GitOps.

Customizing Your Release

While Helm charts are usually installable using default configurations, users will often customize charts with their preferred configuration by overriding the default values. The Helm client allows this by imperatively specifying override values with --set on the command line, and in additional --values files. For example:

helm install my-traefik traefik/traefik --set service.type=ClusterIP

and

helm install my-traefik traefik/traefik --values ci/kind-values.yaml

where ci/kind-values.yaml contains:

service:
 type: ClusterIP

Flux Helm Controller allows these same YAML values overrides on the HelmRelease CRD. These can be declared directly in spec.values:

spec:
 values:
 service:
 type: ClusterIP

and defined in spec.valuesFrom as a list of ConfigMap and Secret resources from which to draw values, allowing reusability and/or greater security. See HelmRelease CRD values overrides documentation for the latest spec.

Managing Secrets and ConfigMaps

You may manage these ConfigMap and Secret resources any way you wish, but there are several benefits to managing these with the Flux Kustomize Controller.

It is fairly straightforward to use Kustomize configMapGenerator to trigger a Helm release upgrade every time the encoded values change. This common use case currently solveable in Helm by adding specially crafted annotations to a chart. The Flux Kustomize Controller method allows you to accomplish this on any chart without additional templated annotations.

You may also use Kustomize Controller built-in Mozilla SOPS integration to securely manage your encrypted secrets stored in git. See the Flux SOPS guide for step-by-step instructions through various use cases.

Automatic Release Upgrades

If you want Helm Controller to automatically upgrade your releases when a new chart version is available in the release’s referenced HelmRepository, you may specify a SemVer range (i.e. >=4.0.0 <5.0.0) instead of a fixed version.

This is useful if your release should use a fixed MAJOR chart version, but want the latest MINOR or PATCH versions as they become available.

For full SemVer range syntax, see Masterminds/semver Checking Version Constraints documentation.

Automatic Uninstalls and Rollback

The Helm Controller offers an extensive set of configuration options to remediate when a Helm release fails, using spec.install.remediation, spec.upgrade.remediation, spec.rollback and spec.uninstall. Features include the option to remediate with an uninstall after an upgrade failure, and the option to keep a failed release for debugging purposes when it has run out of retries.

Here is an example for configuring automated uninstalls (for all available fields, consult the InstallRemediation and Uninstall API references linked above):

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 install:
 # Remediation configuration for when the Helm install
 # (or sequent Helm test) action fails
 remediation:
 # Number of retries that should be attempted on failures before
 # bailing, a negative integer equals to unlimited retries
 retries: -1
 # Configuration options for the Helm uninstall action
 uninstall:
 timeout: 5m
 disableHooks: false
 keepHistory: false

Here is an example of automated rollback configuration (for all available fields, consult the UpgradeRemediation and Rollback API references linked above):

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-release
 namespace: default
spec:
 # ...omitted for brevity
 upgrade:
 # Remediaton configuration for when an Helm upgrade action fails
 remediation:
 # Amount of retries to attempt after a failure,
 # setting this to 0 means no remedation will be
 # attempted
 retries: 5
 # Configuration options for the Helm rollback action
 rollback:
 timeout: 5m
 disableWait: false
 disableHooks: false
 recreate: false
 force: false
 cleanupOnFail: false

Next Steps

Flux: Promote Flux Helm Releases with GitHub Actions

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows how to configure Flux and GitHub Actions to promote Helm Releases across environments when a new Helm chart version is available.

For this guide we assume a scenario with two clusters: staging and production; with the following promotion pipeline:

On the staging cluster, Flux will monitor the Helm repository for new chart versions, and it will automatically upgrade and test the Helm release.
After the Helm release is successfully upgraded, Flux will send an event to GitHub that will trigger a GitHub Actions workflow.
The GitHub workflow receives the new chart version, updates the Flux HelmRelease manifest YAML for the production cluster and opens a Pull Request.
When the Pull Request is merged, Flux upgrades the Helm release on the production cluster to the chart version that was tested in staging.

Prerequisites

For this guide we assume you have two clusters bootstrapped with Flux and a good understanding of how Flux manages Helm releases. Please see the helm example repository to familiarise yourself with Flux and Helm.

Define staging and production releases

For the staging cluster, we’ll define a HelmRelease for which Flux will monitor the Helm repository, and it will automatically upgrade the Helm release to the latest chart version based on a semver range.

Example of clusters/staging/apps/demo.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: demo
 namespace: apps
spec:
 interval: 60m
 chart:
 spec:
 chart: demo
 version: "1.x" # automatically upgrade to the latest version
 interval: 5m # scan the Helm repository every five minutes
 sourceRef:
 kind: HelmRepository
 name: demo-charts
 test:
 enable: true # run tests on upgrades
 valuesFrom:
 - kind: Secret
 name: demo-staging-values

For the production cluster, we’ll define a HelmRelease with a fixed version, the chart version will be updated in Git by GitHub Actions based on the Flux events.

Example of clusters/production/apps/demo.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: demo
 namespace: apps
spec:
 interval: 60m
 chart:
 spec:
 chart: demo
 # This field will be updated by GitHub Actions.
 version: "1.0.0"
 sourceRef:
 kind: HelmRepository
 name: demo-charts
 valuesFrom:
 - kind: Secret
 name: demo-production-values

Define the promotion GitHub workflow

To promote a chart version that was successfully deployed and tested on staging, we’ll create a GitHub workflow that reacts to Flux repository dispatch events.

The event that Flux generate for a HelmRelease upgrade contains metadata about the event, with fields that can be used to determine the chart version that was deployed:

metadata.revision: the Helm chart version deployed.
metadata.oci-digest(optional): the OCI digest of the oci artifact deployed in case of an OCIRepository source.

Promoting a HelmRelease with a chart version

The following .github/workflows/demo-promotion.yaml workflow will react to the Flux repository dispatch events and promote the Helm release to the production cluster.

name: demo-promotion
on:
 repository_dispatch:
 types:
 - HelmRelease/demo.apps

permissions:
 contents: write
 pull-requests: write

jobs:
 promote:
 runs-on: ubuntu-latest
 # Start promotion when the staging cluster has successfully
 # upgraded the Helm release to a new chart version.
 if: |
 github.event.client_payload.metadata.env == 'staging' &&
 github.event.client_payload.severity == 'info'
 steps:
 # Checkout main branch.
 - uses: actions/checkout@v3
 with:
 ref: main
 # Parse the event metadata to determine the chart version deployed on staging.
 - name: Get chart version from staging
 id: staging
 run: |
 VERSION=$(echo ${{ github.event.client_payload.metadata.revision }} | cut -d '@' -f1)
 echo VERSION=${VERSION} >> $GITHUB_OUTPUT
 # Patch the chart version in the production Helm release manifest.
 - name: Set chart version in production
 id: production
 env:
 CHART_VERSION: ${{ steps.staging.outputs.version }}
 run: |
 echo "set chart version to ${CHART_VERSION}"
 yq eval '.spec.chart.spec.version=env(CHART_VERSION)' -i ./clusters/production/apps/demo.yaml
 # Open a Pull Request if an upgrade is needed in production.
 - name: Open promotion PR
 uses: peter-evans/create-pull-request@v4
 with:
 branch: demo-promotion
 delete-branch: true
 token: ${{ secrets.GITHUB_TOKEN }}
 commit-message: Update demo to v${{ steps.staging.outputs.version }}
 title: Promote demo release to v${{ steps.staging.outputs.version }}
 body: |
 Promote demo release on production to v${{ steps.staging.outputs.version }}

The above workflow does the following:

Runs on repository dispatch events issued by Flux with the HelmRelease/demo.apps type.
Filters the events to take into consideration only success Helm release upgrades.
Clones the main branch where the Flux HelmRelease YAML manifests are defined.
Parses the event metadata to determine the chart version deployed on staging.
Patches the chart version in the HelmRelease manifest at clusters/production/apps/demo.yaml.
Creates a new branch called demo-promotion, commits the version change and opens a Pull Request against main.

Note that you should adapt the workflow to match your release name, namespace and YAML path.

Promoting a HelmRelease using OCI digests

The following .github/workflows/demo-promotion.yaml workflow will react to the Flux repository dispatch events and promote the Helm release to the production cluster.

The expected HelmRelease is one with a OCIRepository source.

name: demo-promotion
on:
 repository_dispatch:
 types:
 - HelmRelease/demo.apps

permissions:
 contents: write
 pull-requests: write

jobs:
 promote:
 runs-on: ubuntu-latest
 # Start promotion when the staging cluster has successfully
 # upgraded the Helm release to a new chart version.
 if: |
 github.event.client_payload.metadata.env == 'staging' &&
 github.event.client_payload.severity == 'info'
 steps:
 # Checkout main branch.
 - uses: actions/checkout@v3
 with:
 ref: main
 # Parse the event metadata to determine the chart version deployed on staging.
 - name: Get chart version from staging
 id: staging
 run: |
 DIGEST=$(echo ${{ github.event.client_payload.metadata.oci-digest }} | cut -d '@' -f1)
 echo DIGEST=${DIGEST} >> $GITHUB_OUTPUT
 VERSION=$(echo ${{ github.event.client_payload.metadata.revision }} | cut -d '@' -f1)
 echo VERSION=${VERSION} >> $GITHUB_OUTPUT
 # Patch the digest in the production OCIRepository manifest.
 - name: Set chart version in production
 id: production
 env:
 DIGEST: ${{ steps.staging.outputs.digest }}
 run: |
 echo "set ociRepository digest to ${DIGEST}"
 # This will filter out all resources that do not have a digest field.
 yq eval '(select(.spec.ref.digest) | .spec.ref.digest) = env(DIGEST)' -i ./clusters/production/apps/demo.yaml
 # add the chart version as a line comment
 env lc="version ${{ steps.staging.outputs.version }}" \
 yq eval '(select(.spec.ref.digest) | .spec.ref.digest) line_comment=env(lc)' -i ./clusters/production/apps/demo.yaml
 # Open a Pull Request if an upgrade is needed in production.
 - name: Open promotion PR
 uses: peter-evans/create-pull-request@v4
 with:
 branch: demo-promotion
 delete-branch: true
 token: ${{ secrets.GITHUB_TOKEN }}
 commit-message: Update demo to v${{ steps.staging.outputs.version }} with digest ${{ steps.staging.outputs.digest }}
 title: Promote demo release to v${{ steps.staging.outputs.version }} with digest ${{ steps.staging.outputs.digest }}
 body: |
 Promote demo release on production to v${{ steps.staging.outputs.version }} with digest ${{ steps.staging.outputs.digest }}

The above workflow does the following:

Runs on repository dispatch events issued by Flux with the HelmRelease/demo.apps type.
Filters the events to take into consideration only success Helm release upgrades.
Clones the main branch where the Flux HelmRelease YAML manifests are defined.
Parses the event metadata to determine the chart version deployed on staging and the corresponding OCI digest.
Patches the digest in the OCIRepository manifest at clusters/production/apps/demo.yaml.
Creates a new branch called demo-promotion, commits the version change and opens a Pull Request against main.

Note that you should adapt the workflow to match your release name, namespace and YAML path.

Configure Flux for repository dispatching

On the staging cluster, we’ll configure Flux to send events to GitHub every time it performs a Helm release upgrade.

Example of clusters/staging/apps/demo-github.yaml:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: github
 namespace: apps
spec:
 type: githubdispatch
 address: https://github.com/org/repo
 secretRef:
 name: github-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: demo-dispatch
 namespace: apps
spec:
 providerRef:
 name: github
 summary: "Trigger promotion"
 eventMetadata:
 env: staging
 cluster: staging-1
 region: eu-central-1
 eventSeverity: info
 eventSources:
 - kind: HelmRelease
 name: demo
 inclusionList:
 - ".*.upgrade.*succeeded.*"

Note that you should adapt the above definitions to match your GitHub repository address. If testing is enabled in your HelmRelease, you can use the ".*.test.*succeeded.*" expression in the inclusion list instead of ".*.upgrade.*succeeded.*". This will ensure the promotion happens only after tests have been successfully run.

You also need to create a Kubernetes secret with a GitHub Personal Access Token that has access to the repository:

kubectl -n apps create secret generic github-token \
--from-literal=token=${GITHUB_TOKEN}

GitHub PAT

Note that it is advised to create a dedicated user for Flux under your GitHub organisation. Make the Flux user part of a GitHub team, so that you can give Flux access only to the repositories used with flux bootstrap github.

Relevant documentation

Flux: GitHub Actions Basic App Builder

Mon, 01 Jan 0001 00:00:00 +0000

Disclaimer

This document is under review and may be declared out of scope for Flux.

Note that this guide needs further review in consideration of Flux v2.0.0. It also predates the introduction of OCIRepository and likely needs updates in consideration of those advancements.

Expect this doc to either be archived soon, or to receive some overhaul.

This guide shows how to configure GitHub Actions to build an image for each new commit pushed on a branch, for PRs, or for tags in the most basic way that Flux’s automation can work with and making some considerations for both dev and production.

A single GitHub Actions workflow is presented with a few variations but one simple theme: Flux’s only firm requirement for integrating with CI is for the CI to build and push an image. So this document shows how to do just that.

Scope of this document

Strictly speaking Flux considers CI to be out-of-scope, but this answer frequently leads to bad experiences caused by over-complicated CI built for users who did not firmly grasp the minimum requirements that Flux demands from a supporting CI. This example is intended to cover a majority of use cases with the simplest possible CI workflow.

Users are not expected to strictly adopt this minimum viable solution or view this guidance as strongly prescriptive. You can adapt the example workflow for your use, and you can incorporate Flux’s automation into your dev or production release machinery at a variety of critical points, mostly independent of one another.

We anticipate in this guide that Flux users who are developing one or more apps likely want two build strategies for each app: a Dev build generates a (not semantically versioned) tag from some feature or environment branch with the branch name, commit hash, and timestamp; and a Release build produces a semantic version tag from the release tag that preceded it.

You might want deployment automation in either or both environments, or perhaps neither. This guide shows how to generate image tags in a way that will be ready to work with Flux’s automation for either or both of these scenarios.

How to configure an ImageUpdateAutomation resource to take advantage of Release or Dev builds with automation is covered separately in the Image Update Guide and Sortable image tags guide, respectively.

Example GitHub Actions Workflow

tl;dr: This build workflow does everything that Flux needs. Drop it into .github/workflows/docker-build.yml and reap the benefits.

First copy this example and update IMAGE to point to your own image repository target. Then set DOCKERHUB_USERNAME and DOCKERHUB_TOKEN and you are done. Most git push events will now result in images suitable for Flux to deploy.

For a deeper understanding and some variations, see the remainder of the doc.

name: Docker Build, Push

on:
 push:
 branches:
 - '*'
 tags-ignore:
 - 'release/*'

jobs:
 docker:
 env:
 IMAGE: kingdonb/any_old_app
 runs-on: ubuntu-latest
 steps:
 - name: Prepare
 id: prep
 run: |
 BRANCH=${GITHUB_REF##*/}
 TS=$(date +%s)
 REVISION=${GITHUB_SHA::8}
 BUILD_ID="${BRANCH}-${REVISION}-${TS}"
 LATEST_ID=canary
 if [[ $GITHUB_REF == refs/tags/* ]]; then
 BUILD_ID=${GITHUB_REF/refs\/tags\//}
 LATEST_ID=latest
 fi
 echo BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') >> $GITHUB_OUTPUT
 echo BUILD_ID=${BUILD_ID} >> $GITHUB_OUTPUT
 echo LATEST_ID=${LATEST_ID} >> $GITHUB_OUTPUT


 - name: Set up QEMU
 uses: docker/setup-qemu-action@v3

 - name: Set up Docker Buildx
 uses: docker/setup-buildx-action@v3

 - name: Login to DockerHub
 uses: docker/login-action@v3
 with:
 username: ${{ secrets.DOCKERHUB_USERNAME }}
 password: ${{ secrets.DOCKERHUB_TOKEN }}

 - name: Build and push
 id: docker_build
 uses: docker/build-push-action@v6
 with:
 push: true
 tags: |
 ${{ env.IMAGE }}:${{ steps.prep.outputs.BUILD_ID }}
 ${{ env.IMAGE }}:${{ steps.prep.outputs.LATEST_ID }}

 - name: Image digest
 run: echo ${{ steps.docker_build.outputs.digest }}

This workflow incorporates a few key concepts and properties which are important for Flux.

Workflow Event Triggers

There are two paths through this flow: when a commit is pushed to any branch and when a commit is pushed to any tag, with some exceptions possible as shown with tags-ignore: – this example is given in case you are using the release/* tags as shown in the Jsonnet Render Action example.

These workflows are executed by GitHub Actions on the push event for any branches and tags we specify.

on:
 push:
 branches:
 - '*'
 tags-ignore:
 - 'release/*'

You may want to invert or adjust these patterns depending on how you are using branches, image tags, and git tags. Flux is not prescriptive about any of this. Maybe you only build tags that match a certain pattern, or only commits on the main branch, depending on the need. Some variations are expected and they are out of scope for this guide.

Now let’s walk through the rest of this example workflow.

Docker Build job

The workflow has one job with the id docker whose purpose is to turn commits from push events into deployable images.

An individual image tag name (string) has two parts, IMAGE which represents the image name that is common for all images in the same project, and following that image name separated by a colon is a tag which uniquely identifies a revision of the image. Repositories can hold many tags, and tags can utilize various forms and formats.

Mutable vs. Immutable tags

Image tags can be mutable or immutable. Flux works best with immutable tags: latest and canary are examples of mutable tags.

This example produces both mutable and immutable tags because Flux works with immutable tags, but many users still expect a latest tag even if Flux won’t be able to take advantage of it. Mutable tags are useful for example with environment branches, to stably represent the latest build in a named environment, but their use is generally contrary to GitOps principles. Flux automation demands immutable tags, with a timestamp or something else sortable in the tag string. Thus mutable tags alone are not suitable for most purposes in Flux.

In this example, LATEST_ID represents a mutable tag and latest as a tag represents the last release build that was pushed from any Git tag. The canary tag is the last image that was pushed from any branch.

BUILD_ID represents the immutable tag in both the dev and release path. This is either a literal tag string from Git tag (Flux works best with semver tags) or a ${BRANCH}-${REVISION}-${TS} in this build workflow.

The mutable tags canary and latest are chosen by the script depending on which event triggered the build. If the image is built from a tag, the latest tag is used. If it is built from a branch, canary is used instead. These tags will therefore always point at the “latest” release tag and the latest “canary” however you define it.

This example shows one useful convention among many possible uses for mutable image tags.

Another sensible choice could be to build and push canary images only from the main branch. This script can be as elaborate as you want, the important logic is all contained in the shell script embedded in the Prepare step:

Prepare Step

 steps:
 - name: Prepare
 id: prep
 run: |
 BRANCH=${GITHUB_REF##*/}
 TS=$(date +%s)
 REVISION=${GITHUB_SHA::8}
 BUILD_ID="${BRANCH}-${REVISION}-${TS}"
 LATEST_ID=canary
 if [[ $GITHUB_REF == refs/tags/* ]]; then
 BUILD_ID=${GITHUB_REF/refs\/tags\//}
 LATEST_ID=latest
 fi
 echo BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') >> $GITHUB_OUTPUT
 echo BUILD_ID=${BUILD_ID} >> $GITHUB_OUTPUT
 echo LATEST_ID=${LATEST_ID} >> $GITHUB_OUTPUT

This script has no external effects, it only takes some inputs from environment variables set by GitHub Actions and calculates them into several outputs: BUILD_ID and LATEST_ID. The BUILD_DATE is also exported as an output for informational purposes and is not used elsewhere in the workflow.

TS is the Unix timestamp in seconds, a monotonically increasing value that represents when the build got scheduled. This lets us reliably determine what build is actually latest, even when some builds may take longer or shorter.

Use Immutable Tags

This section highlights another advantage of Flux’s requirement for using timestamped tags instead of a mutable latest tag, in which case the longest build (and not necessarily the latest promoted build) can occasionally win out.

REVISION is the first 8 characters of the GITHUB_SHA, a fingerprint that is kept for humans to differentiate more easily between tags strings that are very similar. It is not meaningful for Flux and can be omitted if preferred. Only TIMESTAMP has any function as it is needed to create an ImagePolicy (reference: Sortable image tags).

Dependencies Setup

These steps prepare the build environment with QEMU and Docker:

 - name: Set up QEMU
# ...
 - name: Set up Docker Buildx

Secrets for your container registry with read and write access can be added in GitHub as Encrypted secrets and retrieved for use when pushing images.

 - name: Login to DockerHub
# ...
 username: ${{ secrets.DOCKERHUB_USERNAME }}
 password: ${{ secrets.DOCKERHUB_TOKEN }}

If the GitHub Container Registry ( GHCR.io) is used, users can skip encrypting secrets and use the write:packages scope with ambient GITHUB_TOKEN instead. The Docker Login action has more specific instructions.

Build and push tag(s)

Now that Docker is logged in, a generic build and push is invoked, pushing both a mutable and an immutable image tag:

 - name: Build and push
 id: docker_build
# ...
 with:
 push: true
 tags: |
 ${{ env.IMAGE }}:${{ steps.prep.outputs.BUILD_ID }}
 ${{ env.IMAGE }}:${{ steps.prep.outputs.LATEST_ID }}

An image digest is printed at the end for information.

 - name: Image digest
 run: echo ${{ steps.docker_build.outputs.digest }}

Flux: GitHub Actions Manifest Generation

Mon, 01 Jan 0001 00:00:00 +0000

Disclaimer

This document is under review and may be declared out of scope for Flux.

Note that this guide no longer describes the recommended way to handle manifest generation in Flux. It also predates the introduction of OCIRepository and needs updates in consideration of those advancements.

Expect this doc to either be archived soon, or to receive a major overhaul in support of the new preferred approach described below.

Author's Note

If you want to use Flux with tooling-generated manifests today, you should capture the output and store it in an OCI Artifact with flux push artifact. The following doc uses an old method that has many disadvantages: it can’t be provenance secured with Cosign. The artifacts can’t be indexed as efficiently as an OCI registry’s tag-based distribution. Nor can the delivery of a private Git repository be authenticated and authorized in a workload cluster, using any cloud-based IAM or ambient environmental credentials that are typically used to secure a private OCI registry.

These methods were developed to bridge the gap for Flux users transitioning from Flux v1. The introduction of the OCIRepository since then, has nigh obsoleted the approach shown here. Please follow the OCI Cheatsheet guide to understand what is possible, and migrate your workflows to use Flux OCI!

This example implements “build-time” manifest generation on GitHub Actions.

Third-party tools are used to generate YAML manifests in a CI job. The updated YAML are committed and pushed to Git, where kustomize-controller finally applies them.

Background

There are many use cases for manifest generation tools, but Flux v2 no longer permits embedding arbitrary binaries with the Flux machinery to run at apply time.

Flux (kustomize-controller) will apply whatever revision of the manifests are at the latest commit, on any branch it is pointed at. By design, Flux doesn’t care for any details of how a commit is generated.

Since “select latest by build time” image automation is deprecated, and since .flux.yaml is also deprecated, some staple workflows are no longer possible without new accommodations from infrastructure.

What Should We Do?

We first recommend users adjust their tagging strategies, which is made clear elsewhere in the docs. This is usually a straightforward adjustment, and enables the use of Image Update Policies; however this may not be feasible or desired in some cases.

Use Manifest Generation

Introducing, Manifest Generation with Jsonnet, for any old app on GitHub!

If you have followed the Flux bootstrap guide and only have one fleet-infra repository, it is recommended to create a separate repository that represents your application for this use case guide, or clone the repository linked above in order to review these code examples which have already been implemented there.

Primary Uses of Flux

Flux’s primary use case for kustomize-controller is to apply YAML manifests from the latest Revision of an Artifact.

Security Consideration

Flux v2 can not be configured to call out to arbitrary binaries that a user might supply with an InitContainer, as it was possible to do in Flux v1.

Motivation for this Guide

In Flux v2 it is assumed if users want to run more than Kustomize with envsubst, that it will be done outside of Flux; the goal of this guide is to show several common use cases of this pattern in secure ways.

Demonstrated Concepts

It is intended, finally, to show through this use case, three fundamental ideas for use in CI to accompany Flux automation:

Writing workflow that can commit changes back to the same branch of a working repository.
A workflow to commit generated content from one directory into a different branch in the repository.
Workflow to commit from any source directory into a target branch on a different repository.

Readers can interpret this document with adaptations for use with other CI providers, or Git source hosts, or manifest generators.

Jsonnet is demonstrated with examples presented in sufficient depth that, hopefully, Flux users who are not already familiar with manifest generation or Jsonnet can pick up kubecfg and start using it to solve novel and interesting configuration problems.

The Choice of GitHub Actions

There are authentication concerns to address with every CI provider and they also differ by Git provider.

Given that GitHub Actions are hosted on GitHub, this guide can be streamlined in some ways. We can almost completely skip configuring authentication. The cross-cutting concern is handled by the CI platform, except in our fourth and final example, the Commit Across Repositories Workflow.

From a GitHub Action, as we must have been authenticated to write to a branch, Workflows also can transitively gain write access to the repo safely.

Mixing and matching from other providers like Bitbucket Cloud, Jenkins, or GitLab will need more attention to these details for auth configurations. GitHub Actions is a platform that is designed to be secure by default.

Manifest Generation Examples

There are several use cases presented.

String Substitution with sed -i
Docker Build and Tag with Version
Jsonnet for YAML Document Rehydration
Commit Across Repositories Workflow

In case these examples are too heavy, this short link guide can help you navigate the four main examples. Finally, the code examples we’ve all been waiting for, the answer to complex .flux.yaml configs in Flux v2! 🎉🎁

String Substitution with `sed -i`

The entry point for these examples begins at .github/workflows/ in any GitHub source repository where your YAML manifests are stored.

GitRepository source only targets one branch

While this first example operates on any branch (branches: ['*']), each Kustomization in Flux only deploys manifests from one branch or tag at a time. Understanding this is key for managing large Flux deployments and clusters with multiple Kustomizations and/or crossing several environments.

First add this directory if needed in your repositories. Find the example below in context, and read on to understand how it works: 01-manifest-generate.yaml.

# ./.github/workflows/01-manifest-generate.yaml
name: Manifest Generation
on:
 push:
 branches:
 - '*'

jobs:
 run:
 name: Push Git Update
 runs-on: ubuntu-latest
 steps:
 - name: Prepare
 id: prep
 run: |
 VERSION=${GITHUB_SHA::8}
 echo BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') >> $GITHUB_OUTPUT
 echo VERSION=${VERSION} >> $GITHUB_OUTPUT

 - name: Checkout repo
 uses: actions/checkout@v3

 - name: Update manifests
 run: ./update-k8s.sh $GITHUB_SHA

 - name: Commit changes
 uses: EndBug/add-and-commit@v7
 with:
 add: '.'
 message: "[ci skip] deploy from ${{ steps.prep.outputs.VERSION }}"
 signoff: true

In the Prepare step, even before the clone, GitHub Actions provides metadata about the commit. Then, Checkout repo performs a shallow clone for the build.

# excerpt from above - set two outputs named "VERSION" and "BUILD_DATE"
VERSION=${GITHUB_SHA::8}
echo BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') >> $GITHUB_OUTPUT
echo VERSION=${VERSION} >> $GITHUB_OUTPUT

When migrating to Flux v2

Users will find that some guidance has changed since Flux v1. Tagging images with a GIT_SHA was a common practice that is no longer supported by Flux’s Image Automation. A newer alternative is adding timestamp or build number in Sortable image tags, preferred by the image-automation-controller.

Next we call out to a shell script update-k8s.sh taking one argument, the Git SHA value from GitHub:

# excerpted from above - run a shell script
- name: Update manifests
 run: ./update-k8s.sh $GITHUB_SHA

That script is below. It performs two in-place string substitutions using sed.

#!/bin/bash

# update-k8s.sh
set -feu # Usage: $0 <GIT_SHA> # Fails when GIT_SHA is not provided

GIT_SHA=${1:0:8}
sed -i "s|image: kingdonb/any-old-app:.*|image: kingdonb/any-old-app:$GIT_SHA|" k8s.yml
sed -i "s|GIT_SHA: .*|GIT_SHA: $GIT_SHA|" flux-config/configmap.yaml

update-k8s.sh receives GITHUB_SHA that the script trims down to 8 characters.

Then, sed -i runs twice, updating k8s.yml and flux-config/configmap.yaml which are also provided as examples here. The new SHA value is added twice, once in each file.

# k8s.yml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: any-old-app
spec:
 replicas: 1
 selector:
 matchLabels:
 app: any-old-app
 template:
 metadata:
 labels:
 app: any-old-app
 spec:
 containers:
 - image: kingdonb/any-old-app:4f314627
 name: any-old-app
---
apiVersion: v1
kind: Service
metadata:
 name: any-old-app
spec:
 type: ClusterIP
 ports:
 - name: "any-old-app"
 port: 3000
 selector:
 app: any-old-app

The convention of including a k8s.yml file in one’s application repository is borrowed from Okteto’s Getting Started Guides, as a simplified example.

The k8s.yml file in the application root is not meant to be applied by Flux, but might be a handy template to keep fresh as a developer reference nonetheless.

The file below, configmap.yaml, is placed in a directory flux-config/ which will be synchronized to the cluster by a Kustomization that we will add in the following step.

# flux-config/configmap.yaml
apiVersion: v1
data:
 GIT_SHA: 4f314627
kind: ConfigMap
metadata:
 creationTimestamp: null
 name: any-old-app-version
 namespace: devl

These are the two files that are re-written in the sed -i example above.

A configmap is an ideal place to write a variable that is needed by any downstream Kustomization, for example to use with envsubst.

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: any-old-app-devl
spec:
 interval: 15m0s
 path: ./
 prune: true
 sourceRef:
 kind: GitRepository
 name: any-old-app-prod
 targetNamespace: prod
 postBuild:
 substituteFrom:
 - kind: ConfigMap
 name: any-old-app-version
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: any-old-app-prod
 namespace: prod
spec:
 interval: 20m0s
 ref:
 branch: deploy
 secretRef:
 name: flux-secret
 url: ssh://git@github.com/kingdonb/csh-flux

Now, any downstream Deployment in the Kustomization can write a PodSpec like this one, to reference the image from the latest commit referenced by the ConfigMap:

# flux/ some-example-deployment.yaml
spec:
 replicas: 1
 selector:
 matchLabels:
 app: any-old-app
 template:
 metadata:
 labels:
 app: any-old-app
 spec:
 containers:
 - image: kingdonb/any-old-app:${GIT_SHA}
 name: any-old-app

Deployment specifications will vary, so adapting this example is left as exercise for the reader. Write it together with a kustomization.yaml, or just add this to a subdirectory anywhere within your Flux Kustomization path.

Docker Build and Tag with Version

Now for another staple workflow: building and pushing an OCI image tag from a Dockerfile in any branch or tag.

From the Actions marketplace, Build and push Docker images provides the heavy lifting in this example. Flux has nothing to do with building images, but we include this still — as some images will need to be built for our use in these examples.

ImageRepository can reflect both branches and tags

This example builds an image for any branch or tag ref and pushes it to Docker Hub. (Note the omission of branches: ['*'] that was in the prior example.) GitHub Secrets DOCKERHUB_USERNAME and DOCKERHUB_TOKEN are used here to authenticate with Docker Hub from within GitHub Actions.

We again borrow a Prepare step from Kustomize Controller’s own release workflow. Find the example below in context, 02-docker-build.yaml, or copy it from below.

# ./.github/workflows/02-docker-build.yaml
name: Docker Build, Push

on:
 push:
 branches:
 - '*'
 tags-ignore:
 - 'release/*'

jobs:
 docker:
 runs-on: ubuntu-latest
 steps:
 - name: Prepare
 id: prep
 run: |
 VERSION=${GITHUB_SHA::8}
 if [[ $GITHUB_REF == refs/tags/* ]]; then
 VERSION=${GITHUB_REF/refs\/tags\//}
 fi
 echo BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') >> $GITHUB_OUTPUT
 echo VERSION=${VERSION} >> $GITHUB_OUTPUT

 - name: Set up QEMU
 uses: docker/setup-qemu-action@v2

 - name: Set up Docker Buildx
 uses: docker/setup-buildx-action@v2

 - name: Login to DockerHub
 uses: docker/login-action@v2
 with:
 username: ${{ secrets.DOCKERHUB_USERNAME }}
 password: ${{ secrets.DOCKERHUB_TOKEN }}

 - name: Build and push
 id: docker_build
 uses: docker/build-push-action@v4
 with:
 push: true
 tags: kingdonb/any-old-app:${{ steps.prep.outputs.VERSION }}

 - name: Image digest
 run: echo ${{ steps.docker_build.outputs.digest }}

The Docker Login Action is used here to enable an authenticated image push.

Any secrets from GitHub Secrets can be used as shown, and support for image registries is explained in the linked README. Add a setting for registry if your app uses any private registry, rather than the implicit Docker Hub registry above.

# for example
with:
registry: registry.cloud.okteto.net

The image tag VERSION comes from the branch or Git tag that triggered the build. Whether that version is a GIT_SHA or a Semantic Version, (or anything in between!) the same workflow can be used to build an OCI image as shown here.

Jsonnet for YAML Document Rehydration

As mentioned before, Flux only monitors one branch or tag per Kustomization.

In the earlier examples, no fixed branch target was specified. Whatever branch triggered the workflow, received the generated YAMLs in the next commit.

If you created your deployment manifests in any branch, the deploy branch or otherwise, it is necessary to add another Kustomization and GitRepository source to apply manifests from that branch and path in the cluster.

In application repositories, it is common to maintain an environment branch, a release branch, or both. Some additional Flux objects may be needed for each new environment target with its own branch. Jsonnet can be used for more easily managing heavyweight repetitive boilerplate configuration such as this.

It is recommended to follow these examples as they are written for better understanding, then later change and adapt them for your own release practices and environments.

GitRepository source only targets one branch

Since Flux uses one branch per Kustomization, to trigger an update we must write to a deploy branch or tag. Even when new app images can come from any branch (eg. for Dev environments where any latest commit is to be deployed) the YAML manifests to deploy will be sourced from just one branch.

It is advisable to protect repository main and release branches with eg. branch policies and review requirements, as through automation, these branches can directly represent the production environment.

The CI user for this example should be allowed to push directly to the deploy branch that Kustomize deploys from; this branch also represents the environment so must be protected in a similar fashion to release.

Only authorized people (and build robots) should be allowed to make writes to a deploy branch.

Jsonnet Render Action

In this example, the outputted YAML manifests, (on successful completion of the Jsonnet render step,) are staged on the deploy branch, then committed and pushed.

The latest commit on the deploy branch is reconciled into the cluster by another Kustomization that is omitted here, as it is assumed that users who have read this far already added this in the previous examples.

You may find the example below in context, 03-release-manifests.yaml, or simply copy it from below.

# ./.github/workflows/03-release-manifests.yaml
name: Build jsonnet
on:
 push:
 tags: ['release/*']
 branches: ['release']

jobs:
 run:
 name: jsonnet push
 runs-on: ubuntu-latest
 steps:
 - name: Prepare
 id: prep
 run: |
 VERSION=${GITHUB_SHA::8}
 if [[ $GITHUB_REF == refs/tags/release/* ]]; then
 VERSION=${GITHUB_REF/refs\/tags\/release\//}
 fi
 echo BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') >> $GITHUB_OUTPUT
 echo VERSION=${VERSION} >> $GITHUB_OUTPUT

 - name: Checkout repo
 uses: actions/checkout@v3

 - name: Setup kubecfg CLI
 uses: kingdonb/kubecfg/action@main

 - name: kubecfg show
 run: kubecfg show manifests/example.jsonnet > output/production.yaml

 - name: Prepare target branch
 run: ./ci/rake.sh deploy

 - name: Commit changes
 uses: EndBug/add-and-commit@v7
 with:
 add: 'production.yaml'
 branch: deploy
 message: "[ci skip] from ${{ steps.prep.outputs.VERSION }}"
 signoff: true

We add three new steps in this example:

# excerpted from above - workflow steps 3, 4, and 5
- name: Setup kubecfg CLI
 uses: kingdonb/kubecfg/action@main

- name: kubecfg show
 run: kubecfg show manifests/example.jsonnet > output/production.yaml

- name: Prepare target branch
 run: ./ci/rake.sh deploy

While the remaining examples will be written to depend on kubecfg, some use cases may prefer to use pure Jsonnet only as it is sandboxed and therefore safer. We plan to use the kubecfg capability to take input from other sources, like variables and references, but also network-driven imports and functions.

# from above - substitute these steps in 03-release-manifests.yaml,
# between "Checkout repo" and "Commit changes" to use plain Jsonnet instead of kubecfg
- id: jsonnet-render
 uses: alexdglover/jsonnet-render@v1
 with:
 file: manifests/example.jsonnet
 output_file: output/production.yaml
 params: dryrun=true;env=prod

- name: Prepare target branch
 run: ./ci/rake.sh deploy

The jsonnet-render step is borrowed from another source, again find it on GitHub Actions Marketplace for more information. For Tanka users, there is also letsbuilders/tanka-action which describes itself as heavily inspired by jsonnet-render.

The EndBug/add-and-commit action is used again

This time, with the help of rake.sh, our change is staged into a different target branch. This is the same deploy branch, regardless of which branch or tag the build comes from; any configured push event can trigger this workflow to trigger an update to the deploy branch.

#!/bin/bash

# ./ci/rake.sh
set -feux # Usage: $0 <BRANCH> # Fails when BRANCH is not provided
BRANCH=$1

# The output/ directory is listed in .gitignore, where jsonnet rendered output.
pushd output

# Fetch git branch 'deploy' and run `git checkout deploy`
/usr/bin/git -c protocol.version=2 fetch \
 --no-tags --prune --progress --no-recurse-submodules \
 --depth=1 origin $BRANCH
git checkout $BRANCH --

# Prepare the output to commit by itself in the deploy branch's root directory.
mv -f ./production.yaml ../ # Overwrite any existing files (no garbage collection here)
git diff

# All done (the commit will take place in the next action!)
popd

Give this file chmod +x before adding and committing; tailor this workflow to your needs. We render from a file manifests/example.jsonnet, it can be anything. The output is a single K8s YAML file, production.yaml.

- name: Commit changes
 uses: EndBug/add-and-commit@v7
 with:
 add: 'production.yaml'
 branch: deploy
 message: "[ci skip] from ${{ steps.prep.outputs.VERSION }}"
 signoff: true

This is Add & Commit with a branch option, to set the target branch. We’ve added a signoff option as well here, to demonstrate another feature of this GitHub Action. There are many ways to use this workflow step. The link provides more information.

The examples that follow can be copied and pasted into manifests/example.jsonnet, then committed to the release branch and pushed to GitHub in order to execute them.

Pushing this to your repository will fail at first, unless and until a deploy branch is created.

Run git checkout --orphan deploy to create a new empty HEAD in your repo.

Run git reset and git stash to dismiss any files that were staged, then run git commit --allow-empty to create an initial empty commit on the branch.

Now you can copy and run these commands to create an empty branch:

#- Add a basic kustomization.yaml file
cat <<EOF > kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- production.yaml
EOF
#- Add a gitignore so output directory is created
cat <<EOF > .gitignore
/output/**
!/output/.keep
EOF
#- Add the .keep file
touch output/.keep
#- Push these files into an empty branch, and go back to main branch
git add output/.keep .gitignore kustomization.yaml
git commit -m'seed deploy kustomization.yaml for prod'
git push -u origin deploy
git checkout main

On the main branch as well, create a .gitignore and /output/.keep for that branch too. We need to make sure that the output/ directory is present for writing whenever the Jsonnet workflow begins. The rake.sh script sweeps files from the output into the root directory of the deploy branch.

Now run git checkout -b release; git push -u origin release to trigger this action and see it working! 🤞🤞

You may need to be sure that GitHub Actions are enabled on your repository before this will work.

Read onward to see some basic as well as more advanced uses of kubecfg.

Jsonnet `ConfigMap` with `envsubst`

The next example “enforces,” or copies, a value from a configMap from one namespace into many namespaces. This is done so that Kustomizations for each namespace can maintain similar config data in their reconciliations while staying DRY, with some configurations that reach across namespace boundaries.

With Jsonnet, Kustomize, and Kustomization Controller’s postBuild which uses envsubst, there are usually a handful of different ways to accomplish the same task.

This example demonstrates a feature called ext_vars or External Variables in the Jsonnet stdlib.

These examples assume (since no such protection has been presented) that nothing prevents production.yaml from writing resources throughout multiple namespaces. This may be difficult or impossible to achieve depending on your environment.

In order to permit a Kustomization to write into different namespaces, some RBAC configuration may be required.

When you write a Kustomization to apply this, be sure you are aware of whether or not you have set targetNamespace on the Flux Kustomization as it may override any namespace settings in the Jsonnet output. You may note similar configuration in the kustomization.yaml we wrote into the deploy branch as described above, in the step for Jsonnet Render Action.

External Variable Substitution

# Any Old App Jsonnet example 0.10.1 - manifests/example.jsonnet

local kube = import 'https://github.com/bitnami-labs/kube-libsonnet/raw/73bf12745b86718083df402e89c6c903edd327d2/kube.libsonnet';

local example = import 'example.libsonnet';

{
 version_configmap: kube.ConfigMap('any-old-app-version') {
 metadata+: {
 namespace: 'prod',
 },
 data+: {
 VERSION: std.extVar('VERSION'),
 },
 },
 flux_kustomization: example.kustomization('any-old-app-prod') {
 metadata+: {
 namespace: 'flux-system',
 },
 spec+: {
 path: './flux-config/',
 postBuild+: {
 substituteFrom+: [
 {
 kind: 'ConfigMap',
 name: 'any-old-app-version',
 },
 ],
 },
 },
 },
 flux_gitrepository: example.gitrepository('any-old-app-prod') {
 metadata+: {
 namespace: 'flux-system',
 },
 spec+: {
 url: 'https://github.com/kingdonb/any_old_app',
 },
 },
}

The above jsonnet declaration example.jsonnet will not complete without its neighbor example.libsonnet (which can be found linked here.) This part of the example contains some boilerplate detail not meant to be copied, like the name any-old-app-prod and the string 'sops-gpg' in decryption.secretRef which should be changed to match your environment).

If you visited the linked example.libsonnet you may have noticed definitions for kustomization and gitrepository that are frankly pretty specific for a library function. They include details you wouldn’t expect to find in a vendor library, like a default git repository URL, and a default hardcoded ref to the name of our Source gitrepository.

This is our library file, so it can have our own implementation-specific details in it if we want to include them. Now, the power of Jsonnet is visible; we get to decide which configuration needs to be exposed in our main example.jsonnet file, and which parameters are defaults provided by the library, that can be treated like boilerplate and re-defined however we want.

data+: {
 VERSION: std.extVar('VERSION'),
},

This is std.extVar from ext_vars mentioned earlier. Arrange for the version to be passed in through the GitHub Actions workflow:

# adapted from above - 03-release-manifests.yaml
- name: kubecfg show
 run: kubecfg show -V VERSION=${{ steps.prep.outputs.VERSION }} manifests/example.jsonnet > output/production.yaml

The neighbor example.libsonnet file contains some boring (but necessary) boilerplate, so that kubecfg can fulfill this jsonnet, to generate and commit the full Kustomize-ready YAML into a deploy branch as specified in the workflow. (The Kustomization for this example is provided from my fleet-infra repo here. Personalize and adapt this for use with your own application or manifest generations.)

The values provided are for example only and should be personalized, or restructured/rewritten completely to suit your preferred template values and instances. For more idiomatic examples written recently by actual Jsonnet pros, the Tanka - Using Jsonnet tutorial is great, and so is Tanka - Parameterizing which I’ll call out specifically for the _config:: object example that is decidedly more elegant than my version of parameter passing.

If you’ve not previously used Jsonnet before, then you might be wondering about that code example you just read and that’s OK! If you have previously used Jsonnet and already know what idiomatic Jsonnet looks like, you might be wondering too… you can probably tell I (author, Kingdon) practically haven’t ever written a lick of Jsonnet before today.

These examples are going to get progressively more advanced as I learn Jsonnet while I go. At this point I already think it’s pretty cool and I barely know how to use it, but I am starting to understand what type of problems people are using it to solve.

ConfigMap values are not treated as secret data, so there is no encryption to contend with; this makes for what seems like a good first example. Jsonnet enthusiasts, please forgive my newness. I am sure that my interpretation of how to write Jsonnet is most likely not optimal or idiomatic.

Above we showed how to pass in a string from our build pipeline, and use it to write back generated Jsonnet manifests into a commit.

Make Two Environments

Here’s a second example, defining two environments in separate namespaces, instead of just one:

# Any Old App Jsonnet example 0.10.2-alpha1 - manifests/example.jsonnet
# Replicate a section of config and change nothing else about it

// ...

{
 version_configmap: kube.ConfigMap('any-old-app-version') {
 metadata+: {
 namespace: 'flux-system',
 },
 data+: {
 VERSION: std.extVar('VERSION'),
 },
 },
 test_flux_kustomization: example.kustomization('any-old-app-test') {
 metadata+: {
 namespace: 'flux-system',
 },
 spec+: {
 path: './flux-config/',
 postBuild+: {
 substituteFrom+: [
 {
 kind: 'ConfigMap',
 name: 'any-old-app-version',
 },
 ],
 },
 targetNamespace: 'test-tenant',
 },
 },
 prod_flux_kustomization: example.kustomization('any-old-app-prod') {
 metadata+: {
 namespace: 'flux-system',
 },
 spec+: {
 path: './flux-config/',
 postBuild+: {
 substituteFrom+: [
 {
 kind: 'ConfigMap',
 name: 'any-old-app-version',
 },
 ],
 },
 targetNamespace: 'prod-tenant',
 },
 },
 flux_gitrepository: example.gitrepository('any-old-app-prod') {
 metadata+: {
 namespace: 'flux-system',
 },
 spec+: {
 url: 'https://github.com/kingdonb/any_old_app',
 },
 },
}

In this example, some front-matter was omitted for brevity. Wait, what? (There’s nothing brief about this example, it’s extra-verbose!)

“I thought Jsonnet was supposed to be DRY.” Be gentle, refactoring is a methodical and deliberate process. We simply copied the original one environment into two environments, test and prod, which differ only in name.

In the next example, we will subtly change one of them to be configured differently from the other.

Change Something and Refactor

Note the string ‘flux-system’ only occurs once now, having been factored into a variable config_ns. These are some basic abstractions in Jsonnet that we can use to start to DRY up our source manifests.

Again, practically nothing changes functionally, this still does exactly the same thing. With another refactoring, we can express this manifest more concisely, thanks to a new library function we can invent, named example.any_old_app.

# Any Old App Jsonnet example 0.10.2-alpha4 - manifests/example.jsonnet
# Make something different between test and prod

// ...

{
 version_configmap: kube.ConfigMap('any-old-app-version') {
 metadata+: {
 namespace: config_ns,
 },
 data+: {
 VERSION: std.extVar('VERSION'),
 },
 },
 test_flux_kustomization: example.any_old_app('test') {
 spec+: {
 prune: true,
 },
 },
 prod_flux_kustomization: example.any_old_app('prod') {
 spec+: {
 prune: false,
 },
 },
 flux_gitrepository: example.gitrepository('any-old-app-prod') {
 metadata+: {
 namespace: config_ns,
 },
 spec+: {
 url: 'https://github.com/kingdonb/any_old_app',
 },
 },
}

Two things have changed to make this refactoring of the config differ from the first version. Hopefully you’ll notice it’s a lot shorter.

Redundant strings have been collapsed into a variable, and more boilerplate has been moved into the library.

This refactored state is perhaps the most obvious to review, and most intentionally clear about its final intent to the reader. Hopefully you noticed the original environments were identical (or if your eyes glossed over because of the wall of values, you’ve at least taken my word for it.)

But now, these two differ. We’re creating two configurations for any_old_app, named test and prod. One of them has prune enabled, the test environment, and prod is set more conservatively to prevent accidental deletions, with a setting of prune: false.

Since the two environments should each differ only by the boolean setting of spec.prune, we can now pack up and hide away the remainder of the config in with the rest of the boilerplate.

Hiding the undifferentiated boilerplate in a library makes it easier to detect and observe this difference in a quick visual review.

Here’s the new library function’s definition:

any_old_app(environment):: self.kustomization('any-old-app-' + environment) {
 metadata+: {
 namespace: 'flux-system',
 },
 spec+: {
 path: './flux-config/',
 postBuild+: {
 substituteFrom+: [
 {
 kind: 'ConfigMap',
 name: 'any-old-app-version',
 },
 ],
 },
 targetNamespace: environment + '-tenant',
 },
},

This excerpt is taken from the 10.2 release version of example.libsonnet, where you can also read the specific definition of kustomization that is invoked with the expression self.kustomization('any-old-app-' + environment).

The evolution of this jsonnet snippet has gone from unnecessarily verbose to perfect redundant clarity. I say redundant, but I’m actually fine with this exactly the way it is. I think, if nothing further changes, we already have met the best way to express this particular manifest with Jsonnet.

But given that config will undoubtedly have to change as the differing requirements of our development teams and their environments grow, this perfect clarity unfortunately can’t last forever in its current form. It will have to scale.

Notice that strings and repeated invocations of any_old_app are written with parallel structure and form, but there’s nothing explicit linking them.

The object-oriented programmer in me can’t help but ask now, “what happens when we need another copy of the environment, this time slightly more different than those two, and … how about two more after that, (and actually, can we really get by with only ten environments?)” — I am inclined towards thinking of this repeated structure as a sign that an object cries out, waiting to be recognized and named, and defined, (and refactored and defined again.)

List Comprehension

So get ready for obfuscated nightmare mode, (which is the name we thoughtfully reserved for the best and final version of the example,) shown below.

# Any Old App Jsonnet example 0.10.2 - manifests/example.jsonnet

// This is a simple manifest generation example to demo some simple tasks that
// can be automated through Flux, with Flux configs rehydrated through Jsonnet.

// This example uses kube.libsonnet from Bitnami. There are other
// Kubernetes libraries available, or write your own!
local kube = import 'https://github.com/bitnami-labs/kube-libsonnet/raw/73bf12745b86718083df402e89c6c903edd327d2/kube.libsonnet';

// The declaration below adds configuration to a more verbose base, defined in
// more detail at the neighbor libsonnet file here:
local example = import 'example.libsonnet';
local kubecfg = import 'kubecfg.libsonnet';
local kustomize = import 'kustomize.libsonnet';

local config_ns = 'flux-system';

local flux_config = [
 kube.ConfigMap('any-old-app-version') {
 data+: {
 VERSION: std.extVar('VERSION'),
 },
 },
 example.gitrepository('any-old-app-prod') {
 spec+: {
 url: 'https://github.com/kingdonb/any_old_app',
 },
 },
] + kubecfg.parseYaml(importstr 'examples/configMap.yaml');

local kustomization = kustomize.applyList([
 kustomize.namespace(config_ns),
]);

local kustomization_output = std.map(kustomization, flux_config);

{ flux_config: kustomization_output } + {

 local items = ['test', 'prod'],

 joined: {
 [ns + '_flux_kustomization']: {
 data: example.any_old_app(ns) {
 spec+: {
 prune: if ns == 'prod' then false else true,
 },
 },
 }
 for ns in items

 // Credit:
 // https://groups.google.com/g/jsonnet/c/ky6sjYj4UZ0/m/d4lZxWbhAAAJ
 // thanks Dave for showing how to do something like this in Jsonnet
 },
}

This is the sixth revision of this example, (some have been omitted from the story, but they are in Git history.) I think it’s really perfect now. If you’re a programmer, I think, this version is perhaps much clearer. That’s why I called it obfuscated nightmare mode, right? (I’m a programmer, I swear.)

The examples/configMap.yaml file can be found in the 0.10.2 tag of kingdonb/any_old_app, it is vestigial and does not serve any functional purpose in this example, except for showing how to compose Jsonnet objects with parsed YAML from a file.

You should note that kubecfg’s kubecfg.parseYaml method always returns an array, even when the importstr input file only contains a single YAML document. Jsonnet arrays, like strings, can be easily added together with a familiar + operator.

Jsonnet objects can also be added to other objects, composing their fields from smaller objects into larger ones. In the example above, we have added the flux_config object to a collection of AnyOldApp objects, a list comprehension from our environments. This is necessary and important because a Jsonnet program or library must always return a single object.

I’m trying to learn Jsonnet as fast as I can, I hope you’re still with me and if not, don’t worry. Where did all of this programming come from? (And what’s a list comprehension?) It really doesn’t matter.

The heavy lifting libraries for this example are from anguslees/kustomize-libsonnet, which implements some basic primitives of Kustomize in Jsonnet. YAML parser is provided by bitnami/kubecfg, and the Jsonnet implementations of Kubernetes primitives by bitnami-labs/kube-libsonnet.

It is a matter of taste whether you consider from above the first, second, or third example to be better stylistically. It is a matter of taste and circumstances, to put a finer point on it. They each have strengths and weaknesses, depending mostly on whatever changes we will have to make to them next.

We can compare these three versions to elucidate the intent of the programmatically most expressive version which followed the other two. If you’re new at this, you may try to explain how these three examples are similar, and also how they differ. Follow the explanation below for added clarity.

If you haven’t studied Jsonnet, this last version may daunt you with its complexity. The fact is YAML is a document store and Jsonnet is a programming language. This complexity is exactly what we came here for, we want our configuration language to be more powerful! Bring on more complex Jsonnet examples!

Breaking It Down

We define a configmap and a gitrepository (in Jsonnet), then put it together with another configmap (from plain YAML). That’s called flux_config.

local kustomization = kustomize.applyList([
 kustomize.namespace(config_ns),
]);

local kustomization_output = std.map(kustomization, flux_config);

This little diddy (above) has the same effect as a Kustomization based on the following instruction to kustomize build, (except it’s all jsonnet):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ${config_ns}

… setting the namespace for all objects in flux_config to the value of config_ns.

Next, we join it together with a list comprehension (at least I think that’s what this is called):

local items = ['test', 'prod'],

joined: {
 [ns + '_flux_kustomization']: {
 data: example.any_old_app(ns) {
 spec+: {
 prune: if ns == 'prod' then false else true,
 },
 },
 }
 for ns in items
},

Two any_old_app templates are invoked programmatically, with different properties and names, and in target namespaces that are based on the environment names. They go on the end of the document list, and Jsonnet renders them alongside of the others, in various namespaces.

This is the same technique as in 01-manifest-generate.yaml, only this time with Jsonnet and kubecfg instead of sed, (and what a difference it makes!)

This is the foundation for some real release machinery for your applications, this is not just a bunch of shell scripts. Whenever any commit hits the release branch, or when any tag in the form release/* is pushed, the repo is configured to push generated manifest changes to a deploy branch.

This behavior is self-contained within the example any_old_app repository in these examples.

We can use GitHub Actions and Jsonnet to populate parameters through ConfigMap values or with extVars, and at the same time, apply Kustomization and GitRepository as new sync infrastructure for the deploy branch with dependencies on those ConfigMaps. The Kustomization refers to the configmap and makes the VERSION or GIT_SHA variable available as a postBuild substitution, with values pulled from that same configmap we just applied.

Later, we can repeat this process with a SOPS encrypted secret.

The process is not very different, though some of the boilerplate is longer, we’ve already learned to pack away boilerplate. Copying and renaming encrypted secrets within the same cluster is possible wherever cluster operators are permitted to both decrypt and encrypt them with the decryption provider.

A credential at spec.decryption.secretRef holds the key for decryption. Without additional configuration secrets can usually be copied freely around the cluster, as it is possible to decrypt them freely anywhere the decryption keys are made available.

Copy `ConfigMap`s

Assume that each namespace will be separately configured as a tenant by itself somehow later, and that each tenant performs its own git reconciliation within the tenant namespace. That config is out of scope for this example. We are only interested in briefly demonstrating some Jsonnet use cases here.

The app version to install is maintained in a ConfigMap in each namespace based on our own decision logic. This can be implemented as a human operator who goes in and updates this variable’s value before release time.

This Jsonnet creates from a list of namespaces, and injects a ConfigMap into each namespace, another example.jsonnet.

# Any Old App Jsonnet example 0.10.3 - manifests/example.jsonnet

local release_config = kube.ConfigMap('any-old-app-version');
local namespace_list = ['prod', 'stg', 'qa', 'uat', 'dev'];

local release_version = '0.10.3';
local latest_candidate = '0.10.3-alpha1';

{
 [ns + '_tenant']: {
 [ns + '_namespace']: {
 namespace: kube.Namespace(ns),
 },
 [ns + '_configmap']: {
 version_data: release_config {
 metadata+: {
 namespace: ns,
 },
 data+: {
 VERSION: if ns == 'prod' || ns == 'stg' then release_version else latest_candidate,
 },
 },
 },
 }
 for ns in namespace_list
}

In this example, we have set up an additional 3 namespaces and assumed that a Flux Kustomization is provided some other way. The deploy configuration of all 5 environments is maintained here, in a single deploy config.

Imagine that two policies should exist for promoting releases into environments. The environments for development, User Acceptance Testing (uat), and Quality Assurance (qa) can all be primed with the latest release candidate build at any given time.

This is perhaps an excessive amount of formality for an open source or cloud-native project, though readers working in regulated environments may recognize this familiar pattern.

This example will possibly fail to apply with the recommended validations enabled, failing with errors that you can review by running flux get kustomization in your flux namespace, like these:

validation failed: namespace/dev created (server dry run)
namespace/prod created (server dry run)
...
Error from server (NotFound): error when creating "14f54b89-2456-4c15-862e-34670dfcda79.yaml": namespaces "dev" not found
Error from server (NotFound): error when creating "14f54b89-2456-4c15-862e-34670dfcda79.yaml": namespaces "prod" not found

As you can perhaps see, the problem is that objects created within the namespace are not valid before creating the namespace. You can disable the validation temporarily by adding a setting validation: none to your Flux Kustomization to get past this error.

In the deployment configuration above, both prod and staging (stg) are kept in sync with the latest release (not pre-release).

Left as an exercise to the reader, we can also ask next: is it possible to supplement this configuration with a Flagger canary so that updates to the production config are able to be manually verified in the staging environment before they are promoted into Production?

(Hint: Look at the Manual Gating feature of Flagger.)

Copy `Secret`s

This example writes the same secretRef into many HelmReleases, to provide for the cluster to be able to use the same imagePullSecret across several Deployments in a namespace. It is a common problem that jsonnet can solve quite handily, without repeating the Secret name over and over as a string.

Because we have decided to create tenants for each namespace, now is a good time to mention flux create tenant.

We can take the output of flux create tenant prod --with-namespace prod --export and use it to create manifests/examples/tenant.yaml. Perhaps in a full implementation, we would create a tenant library function and call it many times to create our tenants.

For this example, you may discard the Namespace and/or ClusterRoleBinding as they are not needed. Here, we actually just need a ServiceAccount to patch.

---
# manifests/examples/tenant.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
 labels:
 toolkit.fluxcd.io/tenant: prod
 name: prod
 namespace: prod

A namespace will be created by our Jsonnet example code instead (or you may comment this line out in the jsonnet below, if you are already working within a tenant.) The ClusterRoleBinding, or a more restrictive RoleBinding, is important for a functioning Flux tenant installation, but it is not needed for this example.

(For more information on multi-tenancy, read the Flux 2 Multi-Tenancy Guide.)

We make an image pull secret with some docker registry credentials, for the purpose of completing the example. This is just for an example, it can be any secret that you want to replicate through several namespaces in the cluster with Jsonnet.

kubectl create secret docker-registry prod-docker-pull-secret \
--namespace=prod \
--docker-username=youruser --docker-password=secretpassword \
--docker-email=required@example.org --dry-run=client -oyaml \
> manifests/examples/sops-image-pull-secret.yaml
sops -e -i manifests/examples/sops-image-pull-secret.yaml

If you are not familiar with SOPS encryption, you should complete the Mozilla SOPS Flux guide before replicating this example, and internalize all of the concepts explained there.

You will need to have configured your cluster’s Kustomization with a decryption provider and decryption keys to enable the inclusion of encrypted secrets in your config repos. It is not safe to write unencrypted secrets into your git repository, and this should be avoided at all costs even if your repository is kept private.

This final Jsonnet example is presented in context as a working reference in the any_old_app repository, once again as example.jsonnet.

# Any Old App Jsonnet example 0.10.4 - manifests/example.jsonnet

local kubecfg = import 'kubecfg.libsonnet';
local kustomize = import 'kustomize.libsonnet';

local tenant =
 kubecfg.parseYaml(importstr 'examples/tenant.yaml');
local pull_secrets =
 kubecfg.parseYaml(importstr 'examples/sops-image-pull-secret.yaml');

local prod_ns = 'prod';
local staging_ns = 'stg';
local image_pull_secret = 'prod-docker-pull-secret';

// Set the Image Pull Secret on each ServiceAccount
local updateConfig(o) = (
 if o.kind == 'ServiceAccount' then o {
 imagePullSecrets: [{ name: image_pull_secret }],
 } else o
);

// Create a namespace, and add to it Namespace and Secret
local prod_tenant = [
 kube.Namespace(prod_ns),
] + pull_secrets + tenant;

// Prod kustomization - apply the updateConfig
local prod_kustomization = kustomize.applyList([
 updateConfig,
]);

// Stg kustomization - apply the updateConfig and "stg" ns
local staging_kustomization = kustomize.applyList([
 updateConfig,
 kustomize.namespace(staging_ns),
]);

// Include both kustomizations in the Jsonnet object
{
 prod: std.map(prod_kustomization, prod_tenant),
 stg: std.map(staging_kustomization, prod_tenant),
}

The kubecfg.parseYaml instruction returns a list of Jsonnet objects. Our own Jsonnet closely mirrors the example provided by anguslees, with a few differences.

The power of kubecfg is well illustrated through this example inspired by anguslees/kustomize-libsonnet. We parse several YAML files and make several minor updates to them. It really doesn’t matter if one document involved is a secret, or that its data is encrypted by SOPS.

If you are coming to Mozilla SOPS support in Flux v2, having used the SealedSecrets controller before when it was recommended in Flux v1, then you are probably surprised that this works. SOPS does not encrypt secret metadata when used with Flux’s Kustomize Controller integration, which makes examples like this one possible.

The ServiceAccount is a part of Flux’s tenant configuration, and a fundamental concept of Kubernetes RBAC. If this concept is still new to you, read more in the Kubernetes docs on Using Service Accounts.

The other fundamental concept to understand is a Namespace.

Secrets are namespaced objects, and ordinary users with tenant privileges cannot reach outside of their namespace. If tenants should manage a Flux Kustomization within their own namespace boundaries, then a sops-gpg secret must be present in the Namespace with the Kustomization. Cross-namespace secret refs are not supported.

However, any principal with access to read a sops-gpg secret can decrypt any data that are encrypted for it.

Each ServiceAccount can list one or more imagePullSecrets, and any pod that binds the ServiceAccount will automatically include any pull secrets provided there. By adding the imagePullSecret to a ServiceAccount, we can streamline including it everywhere that it is needed.

We can apply a list of transformations with kustomize.applyList that provides a list of pass-through mutating functions for Jsonnet to apply to each Jsonnet object; in our case we use the updateConfig function to patch each ServiceAccount with the ImagePullSecret that we want it to use.

Finally, for staging, we additionally apply kustomize.namespace to update all resources to use the stg namespace instead of the prod namespace. The secret can be copied anywhere we want within the reach of our Flux Kustomization, and since our Flux Kustomization still has cluster-admin and local access to the decryption key, there is no obstacle to copying secrets.

Handling `Secret`s

Because a secret is not safe to store in Git unencrypted, Flux recommends using SOPS to encrypt it.

SOPS will produce a different data key for each fresh invocation of sops -e, producing different cipher data even for the same input data. This is true even when the secret content has not changed. This means, unfortunately, it is not practical for a Manifest Generation routine to implement secret transparency without also granting the capability to read secrets to the CI infrastructure.

SOPS stores the metadata required to decrypt each secret in the metadata of the secret, which must be stored unencrypted to allow encrypted secrets to be read by the private key owners.

Secret transparency means that it should be possible for an observer to know when a stored secret has been updated or rotated. Transparency can be achieved in SOPS by running sops as an editor, using sops [encrypted.yaml], which decrypts for editing and re-encrypts the secret upon closing the editor, thereby only changing the cipher text when secret data also changes.

Depending on your access model, this suggestion could be either a complete non-starter, or a helpful add-on.

As an example, Secrets could be read from GitHub Secrets during a CI job, then written encrypted into a secret that is pushed to the deploy branch. This implementation provides a basic solution for simple centralized secrets rotation. But as this would go way beyond simple manifest generation, we consider this beyond the scope of the tutorial, and it is mentioned only as an example of a more complex usage scenario for users to consider.

Replicate `Secrets` Across Namespaces

When the data of a secret is stored in the Git repository, it can be encrypted to store and transmit safely. SOPS in Kustomize supports encryption of only (stringData|data) fields, not secret metadata including namespace. This means that secrets within the same repo can be copied freely and decrypted somewhere else, just as long as the Kustomization still has access to the SOPS private key.

Because of these properties though, copying a SOPS-encrypted secret from one namespace to another within one single Flux tenant is as easy as cloning the YAML manifest and updating the namespace field. Compared to SealedSecrets controller, which does not permit this type of copying; SOPS, on the other hand, does not currently prevent this without some attention being paid to RBAC.

Remember to protect your secrets with RBAC! This is not optional, when handling secrets as in this example.

Protecting `Secrets` from Unauthorized Access

The logical boundary of a secret is any cluster or tenant where the private key is available for decrypting.

This means that any SOPS secret, once encrypted, can be copied anywhere or used as a base for other Kustomizations in the cluster, so long as the Kustomization itself has access to the decryption keys.

It is important to understand that the sops-gpg key that is generated in the Flux SOPS guide can be used by any Kustomization in the flux-system namespace.

It cannot be over-emphasized; if users want secrets to remain secret, the flux-system namespace (and indeed the entire cluster itself) must be hardened and protected, managed by qualified cluster admins. It is recommended that changes which could access encrypted secrets are tightly controlled as much as deemed appropriate.

More Advanced Secrets Usage

The use of KMS as opposed to in-cluster GPG keys with SOPS is left as an exercise for the reader. The basics of KMS with various cloud providers is covered in more depth by the Mozilla SOPS guide.

Another scenario we considered, but rejected for these examples, requires to decrypt and then re-encrypt SOPS secrets, for use with the secretGenerator feature of Kustomize. This workflow is not supported here for reasons already explained.

Flux suggests maintaining the only active copy of the decryption key for a cluster inside of that cluster (though there may be a provision for backups, or some alternate keys permitted to decrypt.) This arrangement makes such use cases significantly more complicated to explain, beyond the scope of this guide.

For those uses though, additional Workflow Actions are provided:

The Decrypt SOPS Secrets action may be useful and it is mentioned here, (but no example uses are provided.)

The Sops Binary Installer action enables more advanced use cases, like encrypting or re-encrypting secrets.

Jsonnet Recap

While much of this type of manipulation could be handled in Kustomization’s postBuild, via envsubst, some configurations are more complicated this way. They can be better handled in CI, where access to additional tools can be provided.

By writing YAML manifests into a Git commit, the same manifests that Kustomize directly applies, they can be saved for posterity. Or projected out into a new pull request where they can be reviewed before application, or with the proper safe-guards in place they can be applied immediately through a more direct-driven automation.

With generated YAML that Flux applies in the cluster directly from Git commits, fui-yoh - that’s GitOps!

Commit Across Repositories Workflow

Flux will not deploy from pushes on just any branch; GitRepository sources target just one specific branch. Merging to a staging branch, for example, can be used to trigger a deployment to a Staging environment.

Manifest generation can be used to solve, broadly, very many problems, such that even with many examples, this guide would never be totally exhaustive.

This is the final example in this guide.

Here we show 🥁 … how to replicate the original behavior of Flux v1’s image automation! 🤯 🎉

You can put this workflow in your application repo, and target it toward your fleet-infra repo.

To replicate the nearest approximation of Flux’s “deploy latest image” feature of yesteryore, we use push events to do the job, as we hinted was possible in an earlier example. This can be done without Flux v1’s redundant and expensive image pull behavior, retrieving build metadata required to order image tags for deployment.

Flux recommends using real version numbers in your image tags, with a canonical ordering.

The alternative is racy and doesn’t always guarantee the latest commit will be the one that is deployed, since this behavior depends on the time that each commit is pushed, and even precisely how long the build takes to complete; the difference is fine for Dev environments, but this is not a strategy for Production use cases.

Your app’s CI can commit and push YAML manifests (or one manifest for each app) into a separate deploy branch for Kustomization to apply. The deploy branch in a separate repository should be a branch to which the CI user is granted write access.

While there are some issues, this is actually perfect for non-prod deployments, eg. in a test environment!

In context, find 04-update-fleet-infra.yaml, or simply copy it from below.

# ./.github/workflows/04-update-fleet-infra.yaml
name: Update Fleet-Infra
on:
 push:
 branches:
 - 'main'

jobs:
 run:
 name: Push Update
 runs-on: ubuntu-latest
 steps:
 - name: Prepare
 id: prep
 run: |
 VERSION=${GITHUB_SHA::8}
 if [[ $GITHUB_REF == refs/tags/* ]]; then
 VERSION=${GITHUB_REF/refs\/tags\//}
 fi
 echo BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') >> $GITHUB_OUTPUT
 echo VERSION=${VERSION} >> $GITHUB_OUTPUT

 - name: Checkout repo
 uses: actions/checkout@v3

 - name: Update manifests
 run: ./update-k8s.sh $GITHUB_SHA

 - name: Push directory to another repository
 uses: cpina/github-action-push-to-another-repository@v1.2

 env:
 API_TOKEN_GITHUB: ${{ secrets.API_TOKEN_GITHUB }}
 with:
 source-directory: 'flux-config'
 destination-github-username: 'kingdonb'
 destination-repository-name: 'fleet-infra'
 target-branch: 'deploy'
 user-email: kingdon+bot@weave.works
 commit-message: "[ci skip] deploy from ${{ steps.prep.outputs.VERSION }}"

This is Push directory to another repository. This is especially useful because Flux v2 is made to work with more than one GitRepository.

If you must use a mono-repo, consider adding a deploy branch to it! There is no need for branches in the same repo to always share a parent and intersect again at a merge point.

A mono-repo can be counter-productive for performance and will create bottlenecks for Flux, as large commits will take longer to clone, and therefore to reconcile. Ignoring with .sourceignore or spec.ignore will unfortunately not help much with this. Some limitations can only be overcome by changing the data structure.

The flux-system is in the main branch of kingdonb/fleet-infra, as is the default. We prepared in advance, an empty commit with no parent in the same repository, on the deploy branch, so that this checkout would begin with an empty workspace that ci/rake.sh could copy the output/ of Jsonnet into.

git checkout --orphan deploy
git reset --hard
git commit --allow-empty -m'initial empty commit'
git push origin deploy

This is not technically regressive when compared to the behavior of Flux v1’s fluxcd.io/automated, actually avoiding image pull depending on push instead to write the latest Git tag, externally and functionally identical to how Flux v1 did automation. Little else is good that we can say about it.

It is a compatibility shim, to bridge the gap for Flux v1 users. If possible, users are encouraged to migrate to using timestamps, build numbers, or semver tags, that are all supported by some Flux v2 image automation features that are still in alpha at the time of this writing.

Flux’s new Image Automation Controllers are the new solution for Production use!

Adapting for Flux v2

In Flux v2, with ImagePolicy, these examples may be adjusted to order tags by their BUILD_DATE, by adding more string information to the tags. Besides a build timestamp, we can also add branch name.

Why not have it all: ${branch}-${sha}-${ts} – this is the suggestion given in:

Example of a build process with timestamp tagging.

Example formats and alternative strings to use for tagging are at:

Sortable image tags to use with automation.

We don’t expect you to follow these examples to the letter. They present an evolution and are meant to show some of the breadth of options that are available, rather than as prescriptive guidance.

If you are on GitHub, and are struggling to get started using GitHub Actions, or maybe still waiting to make a move on your planned migration from Flux v1; we hope that these GitHub Actions examples can help Flux users to better bridge the gap between both versions.

Flux: GitHub Actions Auto Pull Request

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows how to configure GitHub Actions to open a pull request whenever a selected branch is pushed.

From Image Update Guide we saw that Flux can set .spec.git.push.branch to Push updates to a different branch than the one used for checkout.

Configure an ImageUpdateAutomation resource to push to a target branch, where we can imagine some policy dictates that updates must be staged and approved for production before they can be deployed.

kind: ImageUpdateAutomation
metadata:
 name: flux-system
spec:
 git:
 checkout:
 ref:
 branch: main
 push:
 branch: staging

We can show that the automation generates a change in the staging branch which, once the change is approved and merged, gets deployed into production. The image automation is meant to be gated behind a pull request approval workflow, according to policy you may have in place for your repository.

To create the pull request whenever automation creates a new branch, in your manifest repository, add a GitHub Action workflow as below. This workflow watches for the creation of the staging branch and opens a pull request with any desired labels, title text, or pull request body content that you configure.

# ./.github/workflows/staging-auto-pr.yaml
name: Staging Auto-PR
on:
 create:

jobs:
 pull-request:
 runs-on: ubuntu-latest
 if: |
 github.event.ref_type == 'branch' &&
 github.event.ref == 'staging'
 permissions:
 pull-requests: write
 steps:
 - name: Checkout
 uses: actions/checkout@v4
 with:
 fetch-depth: 0
 - name: Create Pull Request
 env:
 GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} # used implicitly by the gh CLI to authenticate with GitHub
 GITHUB_REPO: ${{ github.repository }}
 GITHUB_REF: ${{ github.ref }}
 run: |
 gh pr create \
 --repo=${GITHUB_REPO} \
 --head=staging \
 --base=main \
 --title="Pulling ${GITHUB_REF} into main" \
 --body=":crown: *An automated PR*" \
 --reviewer=kingdonb \
 --draft

In the example above, --head is the source branch and --base is the destination branch.

You can use th workflow above to automatically open a pull request against a destination branch. When staging is merged into the main branch, changes are deployed in production. Be sure to delete the branch after merging so that the workflow runs the next time that the image automation finds something to change, for that you can go to your repository settings and enable the Automatically delete head branches option.

Additional options

The gh pr create CLI command used in the workflow above has more useful options, like --fill-first, --label and --assignee, that setting will help make this workflow more usable. You can assign reviewers, labels, (use markdown emojis in the --body, make variable substitutions in the title, etc.)

This way you can automatically push changes to a staging branch and require review with manual approval of any automatic image updates, before they are applied on your production clusters.

Experiment with these strategies to find the right automated workflow solution for your team!

Flux: Jenkins + Flux

Mon, 01 Jan 0001 00:00:00 +0000

Disclaimer

Note that this guide has not been updated since more than a year ago, it does not address Kubernetes 1.24 or above, and needs to be refreshed.

Expect this doc to either be archived soon, or to receive an overhaul.

This guide explains how to configure Flux with Jenkins, with the core ideas of GitOps Principles in mind. Let Jenkins handle CI (or Continuous Integration: image build and test, tagging and pushing), and let Flux handle CD (or Continuous Deployment) by making use of the Image Update Automation feature.

Declarative Artifacts

In traditional CI/CD systems like Jenkins, the CI infra is often made directly responsible for continuous delivery. Flux treats this arrangement as dangerous, and mitigates this risk by prescribing encapsulation of resources into declarative artifacts, and a strict boundary separating CI from CD.

Jenkins (or any CI system generally speaking) can be cumbersome, and CI builds are an imperative operation that can succeed or fail including sometimes for surprising reasons. Flux obviates any deploy-time link between Jenkins and production deployments, to fulfill the promises of increased reliability and repeatability with GitOps.

Git Manifests

Flux requires YAML manifests to be kept in a git repository or similar artifact hosting, (such as bucket storage which can also be configured to maintain a revision history.) Each revision represents a new “desired state” for the cluster workloads.

Flux represents this as a GitRepository custom resource.

Flux’s Source Controller periodically reconciles the config repository where the cluster’s YAML manifests are maintained, pulling the latest version of the named ref. To invoke the continuous deployment functionality we use a ref which may be updated, such as a branch or tag. Flux deploys updates from whatever commit is at the ref, every spec.interval.

A revision can be any commit at the head of the branch or tag, or a specific commit hash described in the field spec.ref in the GitRepository. We could also specify a semver expression here, so that Flux infers the latest tag within a range specified. There are many possible configurations.

A specific commit hash can also be listed in the GitRepositoryRef, though this is less common than pointing to a branch, as in this case Flux is no longer performing continuous delivery of changes, but instead reconciles with some specific fixed revision as listed and curated by hand. This behavior, though not particularly ergonomic, could be useful during an incident response.

Pinning to a particular revision allows an operator to handle updates explicitly and manually but still via GitOps. While this effectively disables some automated features of Flux, it is also a capable way to operate. Jenkins jobs could also write commit hashes into manifests, and while cats can live with dogs, this is not important for this example.

When adopting Flux’s approach to GitOps, Jenkins should not kubectl apply your manifests anymore. Jenkins does not need direct access to any production clusters. The responsibilities of Jenkins are reduced from a traditional CI/CD system which does both, to only CI. The CD job is delegated entirely to Flux. From the perspective of a production cluster, Jenkins’ responsibility has ended once an image artifact is built and tested, after it has been tagged for deployment and pushed to the Image Repository.

Image Repository

Jenkins is responsible for building and testing OCI images. Those images can be tested internally on Jenkins as part of the build, before pushing, or once deployed in a non-production cluster or other staging context.

Jenkins workflows are often as varied and complex as a snowflake. Finer points of Jenkins image building are not in scope for this guide, but links and some examples are provided to show a strategy that can be used on Jenkins with Flux.

If you’re getting started and maybe have not much experience with Jenkins, (or if perhaps it was the boss who said to use Jenkins,) we hope this information comes in handy.

Jenkins Builds OCI Images

While many companies and people may use Jenkins for building Docker images, it isn’t actually true that Jenkins builds Docker images.

Docker builds OCI images, and Jenkins can shell out to Docker to build images.

Jenkins always uses some other tool to build OCI images. Docker may still be the most commonly reported such tool in use as of this writing, but other tools are growing in popularity like Porter, Buildpacks.io, Earthfile, and many others we have not seen, many which could be used with Jenkins.

One might use declarative pipelines to include the CI configuration in application repos by writing a Jenkinsfile to suit your own needs. We might use the docker plugin, or a privileged pod with HostPath volume to mount /var/run/docker.sock. There are many strategies; this example only shows one way.

Security Concerns

This should attract the attention of InfoSec advocates, as privileged pods and HostPath volumes are generally acknowledged as extremely dangerous.

You should not ever do these things without understanding the risks. Don’t run untrusted code from an unverified source, anywhere near a production cluster. (OK, that’s fine, and we already acknowledged that Jenkins-CI is to be completely separated from production, according to Flux…)

Dockershim was formally deprecated

We might also be running Jenkins on a Kubernetes cluster that doesn’t even have Docker underneath. In this case you may want to use another tool to produce OCI images with Jenkins.

Here, we will use a privileged pod with access to Docker on the host to its most positive effect, by building the image on a single node cluster which is specially earmarked and set aside for Jenkins builds.

This is so that we can leverage Docker’s node-local image storage while building and testing images. When the next stage of the pipeline is executed after an image is built, there is no need to push or pull from a remote image registry as we are on the same machine where the image was built.

Now while this strategy is efficient, it will only work if Kubernetes is running Docker underneath (and that is certainly getting less common now, though it should fortunately remain possible into the future now that Mirantis has taken over support of dockershim.)

CI is out of scope for Flux

Jenkins users will have to find a way of making sense of all this, depending on the details of your situation. None of this is reasonably in scope for Flux to solve, as Flux separates the responsibilities of CI and CD across these well-defined boundaries.

Whatever tool you use for building images, this document aims to explain and show compatible choices that work well with Flux.

It should be clear that if we use Jenkins or Docker, or any other competing tool for building images, much of the same advice for working with Flux will still apply.

What Should We Do?

We recommend users implement SemVer and/or Sortable image tags, to enable the use of Image Update Policies. It is possible for Jenkins to drive deployments with Flux in this way securely through tags, without direct access or explicit coordination with production environment or staging clusters.

GitOps principles suggest that we should manage production workloads as purely declarative artifacts that accurately describe the cluster state in enough detail to reproduce, including version information. Extrapolating the principles, we can also prescribe updates to container images with an automated process, and appropriately constrain this process to only new SemVer releases within a specified range.

Documentation References

Many parts are needed for a complete continuous delivery pipeline with Jenkins and Flux.

The reference below shows how to build and tag Docker images with Jenkins for development environments, for executing tests in a pipeline stage, and lastly tagging a release version for deployment in a production setting.

Testing Infrastructure

We can do testing without an image registry or pushing or pulling any image, because Jenkins builds the image locally on the build node, with a privileged pod that has direct access to Docker on the host. This is significantly faster than pushing before testing.

So (if we’re using a single-node cluster for Jenkins, or by some other tricks perhaps …) the image is created and used with the single node’s Docker daemon.

Those are some assumptions that you may need to check on… anyway, then we tag and push an image any time a new release version was tagged in Git, only after seeing the tests pass.

`jenkinsci/pipeline-examples` repo

Jenkins provides examples of declarative pipelines that use credentials and show how you can use string data elements collected or composed in earlier stages, to drive downstream stages or scripts in different ways, or simply populating environment variables.

Another example executes certain workflow scripts only against a particular branch. We may need to do all of these things, or similar ideas, in order to test and release new versions of our app in production.

For more information on Jenkins pipelines, visit the jenkinsci/pipeline-examples declarative examples.

Example Jenkinsfile

Adapt this if needed, or add this to a project repository with a Dockerfile in its root, as a file Jenkinsfile, and configure a Multibranch Pipeline to trigger when new commits are pushed to any branch or tag.

Find this example in context at kingdonb/jenkins-example-workflow where it is connected with a Jenkins server, and configured to build and push images to docker.io/kingdonb/jenkins-example-workflow.

dockerRepoHost = 'docker.io'
dockerRepoUser = 'kingdonb' // (Username must match the value in jenkinsDockerSecret)
dockerRepoProj = 'jenkins-example-workflow'

// these refer to a Jenkins secret "id", which can be in Jenkins global scope:
jenkinsDockerSecret = 'docker-registry-account'

// blank values that are filled in by pipeline steps below:
gitCommit = ''
branchName = ''
unixTime = ''
developmentTag = ''
releaseTag = ''

pipeline {
 agent {
 kubernetes { yamlFile "jenkins/docker-pod.yaml" }
 }
 stages {
 // Build a Docker image and keep it locally for now
 stage('Build') {
 steps {
 container('docker') {
 script {
 gitCommit = env.GIT_COMMIT.substring(0,8)
 branchName = env.BRANCH_NAME
 unixTime = (new Date().time.intdiv(1000))
 developmentTag = "${branchName}-${gitCommit}-${unixTime}"
 developmentImage = "${dockerRepoUser}/${dockerRepoProj}:${developmentTag}"
 }
 sh "docker build -t ${developmentImage} ./"
 }
 }
 }
 // Push the image to development environment, and run tests in parallel
 stage('Dev') {
 parallel {
 stage('Push Development Tag') {
 when {
 not {
 buildingTag()
 }
 }
 steps {
 withCredentials([[$class: 'UsernamePasswordMultiBinding',
 credentialsId: jenkinsDockerSecret,
 usernameVariable: 'DOCKER_REPO_USER',
 passwordVariable: 'DOCKER_REPO_PASSWORD']]) {
 container('docker') {
 sh """\
 docker login -u \$DOCKER_REPO_USER -p \$DOCKER_REPO_PASSWORD
 docker push ${developmentImage}
 """.stripIndent()
 }
 }
 }
 }
 // Start a second agent to create a pod with the newly built image
 stage('Test') {
 agent {
 kubernetes {
 yaml """\
 apiVersion: v1
 kind: Pod
 spec:
 containers:
 - name: test
 image: ${developmentImage}
 imagePullPolicy: Never
 securityContext:
 runAsUser: 1000
 command:
 - cat
 resources:
 requests:
 memory: 256Mi
 cpu: 50m
 limits:
 memory: 1Gi
 cpu: 1200m
 tty: true
 """.stripIndent()
 }
 }
 options { skipDefaultCheckout(true) }
 steps {
 // Run the tests in the new test container
 container('test') {
 sh (script: "/app/jenkins/run-tests.sh")
 }
 }
 }
 }
 }
 stage('Push Release Tag') {
 when {
 buildingTag()
 }
 steps {
 script {
 releaseTag = env.TAG_NAME
 releaseImage = "${dockerRepoUser}/${dockerRepoProj}:${releaseTag}"
 }
 container('docker') {
 withCredentials([[$class: 'UsernamePasswordMultiBinding',
 credentialsId: jenkinsDockerSecret,
 usernameVariable: 'DOCKER_REPO_USER',
 passwordVariable: 'DOCKER_REPO_PASSWORD']]) {
 sh """\
 docker login -u \$DOCKER_REPO_USER -p \$DOCKER_REPO_PASSWORD
 docker tag ${developmentImage} ${releaseImage}
 docker push ${releaseImage}
 """.stripIndent()
 }
 }
 }
 }
 }
}

The example above should do the necessary work for building, tagging, and pushing images for development and production.

Instructions for Use

Fork the repo from kingdonb/jenkins-example-workflow, to see the other details like a basic suitable example Dockerfile, the jenkins/docker-pod.yaml, and the jenkins/run-tests.sh script.

Create a Multibranch Pipeline and associate it with your repository. Configure it to build commits from at least one branch (or all branches) and any tags.

Development Image

When you push a commit to a branch, it will be built and tested locally on the Jenkins node, and also a dev image is pushed in parallel to the image repository.

This image is tagged with a Sortable image tag of the format {branch}-{sha}-{ts}.

This can be deployed automatically by Flux, with a more relaxed policy for development environments. There is no requirement that tests must pass in order to deploy the latest image in development. You can add such a requirement later, or change this in any way that makes sense for your use case.

Release `SemVer` Image

We will confirm the tests are passing before pushing any production tag, in the Push Release Tag stage, which only runs after our Test stage has succeeded.

The corresponding Flux resource is covered in the using an ImagePolicy object section of the aforementioned guide.

When you push a git tag, different workflow is used because of:

when {
buildingTag()
}

and

when {
not {
buildingTag()
}
}

This is important since git tags can be used for Automating image updates to Git in production.

Using SemVer tags, you can automatically promote new tags to production via policy.

In this guide we assume you will manually create and push Git tags whenever needed, then promote them through the pipeline. These builds will run docker build again for the tag, which should hit the cache and complete quickly, then run tests again, and finally on success push a SemVer image tag.

Example Resources

You can find this example tested with a Jenkins instance at kingdonb/jenkins-example-workflow; it has been configured to run as you can see from the commit status checks found on all commits, tags, and pull requests. The images are pushed to docker.io/kingdonb/jenkins-example-workflow.

Configuration of webhooks would be a logical next step if not already configured and working, so that Jenkins can trigger builds without polling or manual intervention by an operator. Demonstrating this advanced CI feature of Jenkins is again something that is out of scope for this guide.

Wrap Up

By pushing image tags, Jenkins can update the cluster using Flux’s pull-based model for updating. When a new image is pushed that has a newer image tag, and meets the filters of the configured policy, Flux is made aware of it by Image Reflector API, which captures the new candidate tag as an Image resource.

Your CI workflow can be based on these examples, or may turn out completely different. Jenkins has a rich ecosystem of plugins, and the Jenkinsfile is as diverse and powerful as any programming language. If you are using Jenkins already, you probably already know exactly how you want it to build images.

If you are concerned about running Docker and Kubernetes together, or if you need to use these workflows in a cluster that cannot run container images as root, or in privileged mode, for an alternative build strategy that can still work with Jenkins in rootless mode, we recommend you check out Kubernetes examples for Buildkit and the Buildkit CLI for Kubectl.

These were recently presented together at KubeCon/CloudNativeCon EU 2021.

The finer points of building OCI images in Jenkins are out of scope for this guide. These examples are meant to be kept simple, though complete, and we refrain from sharing strong opinions about how CI should work here, because it’s simply out of scope for Flux to weigh in about these topics. We meant to show some ways that Jenkins CI, or any similar functioning tool, can be used with Flux.

This works without Jenkins connecting to production clusters, only building images, and Flux only receives published image tags. So there is no strong coupling or inter-dependency between CI and CD!

Update deployments via Flux’s ImagePolicy CRD, and the Image Update Automation API.

Flux – Use Cases

Flux: Running pre and post-deployment jobs with Flux

Repository structure

Configure the deployment pipeline

Flux: Karmada + Flux

Disclaimer

Background

Karmada Setup

Flux Installation

Tip

Helm release propagation

Customize the Helm release for specific clusters

Flux: Flux for Helm Users

What Does Flux add to Helm?

Getting Started

Customizing Your Release

Managing Secrets and ConfigMaps

Automatic Release Upgrades

Automatic Uninstalls and Rollback

Next Steps

Flux: Promote Flux Helm Releases with GitHub Actions

Prerequisites

Define staging and production releases

Define the promotion GitHub workflow

Promoting a HelmRelease with a chart version

Promoting a HelmRelease using OCI digests

Configure Flux for repository dispatching

GitHub PAT

Relevant documentation

Flux: GitHub Actions Basic App Builder

Disclaimer

Scope of this document

Example GitHub Actions Workflow

Workflow Event Triggers

Docker Build job

Mutable vs. Immutable tags

Prepare Step

Use Immutable Tags

Dependencies Setup

DockerHub Login

Build and push tag(s)

Further Reading on CI

Image Provenance Security

Caching for Fast Builds

Flux: GitHub Actions Manifest Generation

Disclaimer

Author's Note

Background

What Should We Do?

Use Manifest Generation

Primary Uses of Flux

Security Consideration

Motivation for this Guide

Demonstrated Concepts

The Choice of GitHub Actions

Manifest Generation Examples

String Substitution with sed -i

GitRepository source only targets one branch

When migrating to Flux v2

Docker Build and Tag with Version

ImageRepository can reflect both branches and tags

Jsonnet for YAML Document Rehydration

GitRepository source only targets one branch

Jsonnet Render Action

The EndBug/add-and-commit action is used again

Jsonnet ConfigMap with envsubst

External Variable Substitution

Make Two Environments

Change Something and Refactor

List Comprehension

Breaking It Down

Copy ConfigMaps

Copy Secrets

Handling Secrets

Replicate Secrets Across Namespaces

Protecting Secrets from Unauthorized Access

More Advanced Secrets Usage

Jsonnet Recap

Commit Across Repositories Workflow

Adapting for Flux v2

String Substitution with `sed -i`

Jsonnet `ConfigMap` with `envsubst`

Copy `ConfigMap`s

Copy `Secret`s

Handling `Secret`s

Replicate `Secrets` Across Namespaces

Protecting `Secrets` from Unauthorized Access

`jenkinsci/pipeline-examples` repo

Release `SemVer` Image