Flux – Dev

Flagger: Development Guide

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to build, test and run Flagger from source.

Setup dev environment

Flagger is written in Go and uses Go modules for dependency management.

On your dev machine install the following tools:

go >= 1.18
git >;= 2.20
bash >= 5.0
make >= 3.81
kubectl >= 1.22
kustomize >= 4.4
helm >= 3.0
docker >= 19.03

You’ll also need a Kubernetes cluster for testing Flagger. You can use Minikube, Kind, Docker desktop or any remote cluster (AKS/EKS/GKE/etc) Kubernetes version 1.22 or newer.

To start contributing to Flagger, fork the repository on GitHub.

Create a dir inside your GOPATH:

mkdir -p $GOPATH/src/github.com/fluxcd

Clone your fork:

cd $GOPATH/src/github.com/fluxcd
git clone https://github.com/YOUR_USERNAME/flagger
cd flagger

Set Flagger repository as upstream:

git remote add upstream https://github.com/fluxcd/flagger.git

Sync your fork regularly to keep it up-to-date with upstream:

git fetch upstream
git checkout main
git merge upstream/main

Build

Download Go modules:

go mod download

Build Flagger binary:

make build

Build load tester binary:

make loadtester-build

Code changes

We require all commits to be signed. By signing off with your signature, you certify that you wrote the patch or otherwise have the right to contribute the material by the rules of the DCO.

If your user.name and user.email are configured in your Git config, you can sign your commit automatically with:

git commit -s

Before submitting a PR, make sure your changes are covered by unit tests.

If you made changes to go.mod run:

go mod tidy

If you made changes to pkg/apis regenerate Kubernetes client sets with:

make codegen

Run code formatters:

go install golang.org/x/tools/cmd/goimports@latest

make fmt

Run unit tests:

make test

API changes

If you made changes to pkg/apis regenerate the Kubernetes client sets with:

make codegen

Update the validation spec in artifacts/flagger/crd.yaml and run:

make crd

Note that any change to the CRDs must be accompanied by an update to the Open API schema.

Manual testing

Install a service mesh and/or an ingress controller on your cluster and deploy Flagger using one of the install options listed here.

If you made changes to the CRDs, apply your local copy with:

kubectl apply -f artifacts/flagger/crd.yaml

Shutdown the Flagger instance installed on your cluster (replace the namespace with your mesh/ingress one):

kubectl -n istio-system scale deployment/flagger --replicas=0

Port forward to your Prometheus instance:

kubectl -n istio-system port-forward svc/prometheus 9090:9090

Run Flagger locally against your remote cluster by specifying a kubeconfig path:

go run cmd/flagger/ -kubeconfig=$HOME/.kube/config \
-log-level=info \
-mesh-provider=istio \
-metrics-server=http://localhost:9090

Another option to manually test your changes is to build and push the image to your container registry:

make build
docker build -t <YOUR-DOCKERHUB-USERNAME>/flagger:<YOUR-TAG> .
docker push <YOUR-DOCKERHUB-USERNAME>/flagger:<YOUR-TAG>

Deploy your image on the cluster and scale up Flagger:

kubectl -n istio-system set image deployment/flagger flagger=<YOUR-DOCKERHUB-USERNAME>/flagger:<YOUR-TAG>
kubectl -n istio-system scale deployment/flagger --replicas=1

Now you can use one of the tutorials to manually test your changes.

Integration testing

Flagger end-to-end tests can be run locally with Kubernetes Kind.

Create a Kind cluster:

kind create cluster

Build Flagger container image and load it on the cluster:

make build
docker build -t test/flagger:latest .
kind load docker-image test/flagger:latest

Run the Istio e2e tests:

./test/istio/run.sh

For each service mesh and ingress controller, there is a dedicated e2e test suite, choose one that matches your changes from this list.

When you open a pull request on Flagger repo, the unit and integration tests will be run in CI.

Flagger: Release Guide

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to release Flagger.

Release

Flagger

To release a new Flagger version (e.g. 2.0.0) follow these steps:

create a branch git checkout -b release-2.0.0
set the version in code and manifests TAG=2.0.0 make version-set
commit changes and merge PR
checkout main git checkout main && git pull
tag main make release

Flagger load tester

To release a new Flagger load tester version (e.g. 2.0.0) follow these steps:

create a branch git checkout -b release-ld-2.0.0
set the version in code (cmd/loadtester/main.go#VERSION)
set the version in the Helm chart (charts/loadtester/Chart.yaml and values.yaml)
set the version in manifests (kustomize/tester/deployment.yaml)
commit changes and push the branch upstream
in GitHub UI, navigate to Actions and run the push-ld workflow selecting the release branch
after the workflow finishes, open the PR which will run the e2e tests using the new tester version
merge the PR if the tests pass

CI

After the tag has been pushed to GitHub, the CI release pipeline does the following:

creates a GitHub release
pushes the Flagger binary and change log to GitHub release
pushes the Flagger container image to GitHub Container Registry
pushed the Flagger install manifests to GitHub Container Registry
signs all OCI artifacts and release assets with Cosign and GitHub OIDC
pushes the Helm chart to github-pages branch
GitHub pages publishes the new chart version on the Helm repository

Docs

The documentation website is built from the docs branch.

After a Flagger release, publish the docs with:

git checkout main && git pull
git checkout docs
git rebase main
git push origin docs

Lastly open a PR with all the docs changes on fluxcd/website to update fluxcd.io/flagger.

Flagger: Upgrade Guide

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to upgrade Flagger.

Upgrade canaries v1alpha3 to v1beta1

Canary CRD changes in canaries.flagger.app/v1beta1:

the spec.canaryAnalysis field has been deprecated and replaced with spec.analysis
the spec.analysis.interval and spec.analysis.threshold fields are required
the status.lastAppliedSpec and status.lastPromotedSpec hashing algorithm changed to hash/fnv
the spec.analysis.alerts array can reference alertproviders.flagger.app/v1beta1 resources
the spec.analysis.metrics[].templateRef can reference a metrictemplate.flagger.app/v1beta1 resource
the metric.threshold field has been deprecated and replaced with metric.thresholdRange
the metric.query field has been deprecated and replaced with metric.templateRef
the spec.ingressRef.apiVersion accepts networking.k8s.io/v1beta1
the spec.targetRef can reference DaemonSet kind
the spec.service.meshName field has been deprecated and no longer used for provider: appmesh:v1beta2

Upgrade procedure:

install the v1beta1 CRDs
update Flagger deployment
replace apiVersion: flagger.app/v1alpha3 with apiVersion: flagger.app/v1beta1 in all canary manifests
replace spec.canaryAnalysis with spec.analysis in all canary manifests
update canary manifests in cluster

Note that after upgrading Flagger, all canaries will be triggered as the hash value used for tracking changes is computed differently. You can set spec.skipAnalysis: true in all canary manifests before upgrading Flagger, do the upgrade, wait for Flagger to finish the no-op promotions and finally set skipAnalysis to false.

Update builtin metrics:

replace threshold with thresholdRange.min for request-success-rate
replace threshold with thresholdRange.max for request-duration

metrics:
- name: request-success-rate
 thresholdRange:
 min: 99
 interval: 1m
- name: request-duration
 thresholdRange:
 max: 500
 interval: 1m

Istio telemetry v2

Istio 1.5 comes with a breaking change for Flagger uses. In Istio telemetry v2 the metric istio_request_duration_seconds_bucket has been removed and replaced with istio_request_duration_milliseconds_bucket and this breaks the request-duration metric check.

If are using Istio 1.4, you can create a metric template using the old duration metric like this:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: latency
 namespace: istio-system
spec:
 provider:
 type: prometheus
 address: http://prometheus.istio-system:9090
 query: |
 histogram_quantile(
 0.99,
 sum(
 rate(
 istio_request_duration_seconds_bucket{
 reporter="destination",
 destination_workload_namespace="{{ namespace }}",
 destination_workload=~"{{ target }}"
 }[{{ interval }}]
 )
 ) by (le)
 )

In the canary manifests, replace the request-duration metric with latency:

metrics:
- name: latency
 templateRef:
 name: latency
 namespace: istio-system
 thresholdRange:
 max: 0.500
 interval: 1m