Flux – Tutorials

Flagger: Istio Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use Istio and Flagger to automate canary deployments.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Istio v1.5 or newer.

Install Istio with telemetry support and Prometheus:

istioctl manifest install --set profile=default

# Suggestion: Please change release-1.8 in below command, to your real istio version.
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.8/samples/addons/prometheus.yaml

Install Flagger in the istio-system namespace:

kubectl apply -k github.com/fluxcd/flagger//kustomize/istio

Create an ingress gateway to expose the demo app outside of the mesh:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: public-gateway
 namespace: istio-system
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - "*"

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services, Istio destination rules and virtual services). These objects expose the application inside the mesh and drive the canary analysis and promotion.

Create a test namespace with Istio sidecar injection enabled:

kubectl create ns test
kubectl label namespace test istio-injection=enabled

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a canary custom resource (replace example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # service port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 # Istio gateways (optional)
 gateways:
 - public-gateway.istio-system.svc.cluster.local
 # Istio virtual service host names (optional)
 hosts:
 - app.example.com
 # Istio traffic policy (optional)
 trafficPolicy:
 tls:
 # use ISTIO_MUTUAL when mTLS is enabled
 mode: DISABLE
 # Istio retry policy (optional)
 retries:
 attempts: 3
 perTryTimeout: 1s
 retryOn: "gateway-error,connect-failure,refused-stream"
 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 10
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"

Note that when using Istio 1.4 you have to replace the request-duration with a metric template.

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary. The canary analysis will run for five minutes while validating the HTTP metrics and rollout hooks every minute.

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
destinationrule.networking.istio.io/podinfo-canary
destinationrule.networking.istio.io/podinfo-primary
virtualservice.networking.istio.io/podinfo

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Scaling up podinfo.test
 Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 2m flagger Advance podinfo.test canary weight 20
 Normal Synced 2m flagger Advance podinfo.test canary weight 25
 Normal Synced 1m flagger Advance podinfo.test canary weight 30
 Normal Synced 1m flagger Advance podinfo.test canary weight 35
 Normal Synced 55s flagger Advance podinfo.test canary weight 40
 Normal Synced 45s flagger Advance podinfo.test canary weight 45
 Normal Synced 35s flagger Advance podinfo.test canary weight 50
 Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
 Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

A canary deployment is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps mounted as volumes or mapped to environment variables
Secrets mounted as volumes or mapped to environment variables

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-01-16T14:05:07Z
prod frontend Succeeded 0 2019-01-15T16:15:07Z
prod backend Failed 0 2019-01-14T17:05:07Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses the rollout.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it flagger-loadtester-xx-xx sh

Generate HTTP 500 errors:

watch curl http://podinfo-canary:9898/status/500

Generate latency:

watch curl http://podinfo-canary:9898/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 10
 Phase: Failed
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger Starting canary deployment for podinfo.test
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 55.06% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 47.00% < 99%
 Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement success rate 38.08% < 99%
 Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
 Warning Synced 1m flagger Canary failed! Scaling down podinfo.test

Session Affinity

While Flagger can perform weighted routing and A/B testing individually, with Istio it can combine the two leading to a Canary release with session affinity. For more information you can read the deployment strategies docs.

Create a canary custom resource (replace app.example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # service port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 # Istio gateways (optional)
 gateways:
 - public-gateway.istio-system.svc.cluster.local
 # Istio virtual service host names (optional)
 hosts:
 - app.example.com
 # Istio traffic policy (optional)
 trafficPolicy:
 tls:
 # use ISTIO_MUTUAL when mTLS is enabled
 mode: DISABLE
 # Istio retry policy (optional)
 retries:
 attempts: 3
 perTryTimeout: 1s
 retryOn: "gateway-error,connect-failure,refused-stream"
 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 10
 # session affinity config
 sessionAffinity:
 # name of the cookie used
 cookieName: flagger-cookie
 # max age of the cookie (in seconds)
 # optional; defaults to 86400
 maxAge: 21600
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"

Save the above resource as podinfo-canary-session-affinity.yaml and then apply it:

kubectl apply -f ./podinfo-canary-session-affinity.yaml

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

You can load app.example.com in your browser and refresh it until you see the requests being served by podinfo:6.0.1. All subsequent requests after that will be served by podinfo:6.0.1 and not podinfo:6.0.0 because of the session affinity configured by Flagger with Istio.

Traffic mirroring

For applications that perform read operations, Flagger can be configured to drive canary releases with traffic mirroring. Istio traffic mirroring will copy each incoming request, sending one request to the primary and one to the canary service. The response from the primary is sent back to the user and the response from the canary is discarded. Metrics are collected on both requests so that the deployment will only proceed if the canary metrics are within the threshold values.

Note that mirroring should be used for requests that are idempotent or capable of being processed twice (once by the primary and once by the canary).

You can enable mirroring by replacing stepWeight/maxWeight with iterations and by setting analysis.mirror to true:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 analysis:
 # schedule interval
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 5
 # total number of iterations
 iterations: 10
 # enable traffic shadowing 
 mirror: true
 # weight of the traffic mirrored to your canary (defaults to 100%)
 mirrorWeight: 100
 metrics:
 - name: request-success-rate
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 thresholdRange:
 max: 500
 interval: 1m
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo.test:9898/"

With the above configuration, Flagger will run a canary release with the following steps:

detect new revision (deployment spec, secrets or configmaps changes)
scale from zero the canary deployment
wait for the HPA to set the canary minimum replicas
check canary pods health
run the acceptance tests
abort the canary release if tests fail
start the load tests
mirror 100% of the traffic from primary to canary
check request success rate and request duration every minute
abort the canary release if the metrics check failure threshold is reached
stop traffic mirroring after the number of iterations is reached
route live traffic to the canary pods
promote the canary (update the primary secrets, configmaps and deployment spec)
wait for the primary deployment rollout to finish
wait for the HPA to set the primary minimum replicas
check primary pods health
switch live traffic back to primary
scale to zero the canary
send notification with the canary analysis result

The above procedure can be extended with custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: Istio A/B Testing

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to automate A/B testing with Istio and Flagger.

Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions. In an A/B testing scenario, you’ll be using HTTP headers or cookies to target a certain segment of your users. This is particularly useful for frontend applications that require session affinity.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Istio v1.0 or newer.

Install Istio with telemetry support and Prometheus:

istioctl manifest install --set profile=default

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.8/samples/addons/prometheus.yaml

Install Flagger in the istio-system namespace:

kubectl apply -k github.com/fluxcd/flagger//kustomize/istio

Create an ingress gateway to expose the demo app outside of the mesh:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
 name: public-gateway
 namespace: istio-system
spec:
 selector:
 istio: ingressgateway
 servers:
 - port:
 number: 80
 name: http
 protocol: HTTP
 hosts:
 - "*"

Bootstrap

Create a test namespace with Istio sidecar injection enabled:

kubectl create ns test
kubectl label namespace test istio-injection=enabled

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a canary custom resource (replace example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # container port
 port: 9898
 # Istio gateways (optional)
 gateways:
 - public-gateway.istio-system.svc.cluster.local
 # Istio virtual service host names (optional)
 hosts:
 - app.example.com
 # Istio traffic policy (optional)
 trafficPolicy:
 tls:
 # use ISTIO_MUTUAL when mTLS is enabled
 mode: DISABLE
 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # total number of iterations
 iterations: 10
 # max number of failed iterations before rollback
 threshold: 2
 # canary match condition
 match:
 - headers:
 user-agent:
 regex: ".*Firefox.*"
 - headers:
 cookie:
 regex: "^(.*?;)?(type=insider)(;.*)?$"
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # generate traffic during analysis
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 15s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 -H 'Cookie: type=insider' http://podinfo.test:9898/"

Note that when using Istio 1.5 you have to replace the request-duration with a metric template.

The above configuration will run an analysis for ten minutes targeting Firefox users and those that have an insider cookie.

Save the above resource as podinfo-abtest.yaml and then apply it:

kubectl apply -f ./podinfo-abtest.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
destinationrule.networking.istio.io/podinfo-canary
destinationrule.networking.istio.io/podinfo-primary
virtualservice.networking.istio.io/podinfo

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/abtest

Status:
 Failed Checks: 0
 Phase: Succeeded
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Scaling up podinfo.test
 Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
 Normal Synced 2m flagger Advance podinfo.test canary iteration 4/10
 Normal Synced 2m flagger Advance podinfo.test canary iteration 5/10
 Normal Synced 1m flagger Advance podinfo.test canary iteration 6/10
 Normal Synced 1m flagger Advance podinfo.test canary iteration 7/10
 Normal Synced 55s flagger Advance podinfo.test canary iteration 8/10
 Normal Synced 45s flagger Advance podinfo.test canary iteration 9/10
 Normal Synced 35s flagger Advance podinfo.test canary iteration 10/10
 Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
 Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 100 2019-03-16T14:05:07Z
prod frontend Succeeded 0 2019-03-15T16:15:07Z
prod backend Failed 0 2019-03-14T17:05:07Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors and high latency to test Flagger’s rollback.

Generate HTTP 500 errors:

watch curl -b 'type=insider' http://app.example.com/status/500

Generate latency:

watch curl -b 'type=insider' http://app.example.com/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Failed Checks: 2
 Phase: Failed
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger Starting canary deployment for podinfo.test
 Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
 Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
 Warning Synced 2m flagger Rolling back podinfo.test failed checks threshold reached 2
 Warning Synced 1m flagger Canary failed! Scaling down podinfo.test

The above procedure can be extended with custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: Apache APISIX Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use the Apache APISIX and Flagger to automate canary deployments.

Prerequisites

Flagger requires a Kubernetes cluster v1.19 or newer and Apache APISIX v2.15 or newer and Apache APISIX Ingress Controller v1.5.0 or newer.

Install Apache APISIX and Apache APISIX Ingress Controller with Helm v3:

helm repo add apisix https://charts.apiseven.com
kubectl create ns apisix

helm upgrade -i apisix apisix/apisix --version=0.11.3 \
--namespace apisix \
--set apisix.podAnnotations."prometheus\.io/scrape"=true \
--set apisix.podAnnotations."prometheus\.io/port"=9091 \
--set apisix.podAnnotations."prometheus\.io/path"=/apisix/prometheus/metrics \
--set pluginAttrs.prometheus.export_addr.ip=0.0.0.0 \
--set pluginAttrs.prometheus.export_addr.port=9091 \
--set pluginAttrs.prometheus.export_uri=/apisix/prometheus/metrics \
--set pluginAttrs.prometheus.metric_prefix=apisix_ \
--set ingress-controller.enabled=true \
--set ingress-controller.config.apisix.serviceNamespace=apisix

Install Flagger and the Prometheus add-on in the same namespace as Apache APISIX:

helm repo add flagger https://flagger.app

helm upgrade -i flagger flagger/flagger \
--namespace apisix \
--set prometheus.install=true \
--set meshProvider=apisix

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and an ApisixRoute). These objects expose the application outside the cluster and drive the canary analysis and promotion.

Create a test namespace:

kubectl create ns test

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test

Create an Apache APISIX ApisixRoute, Flagger will reference and generate the canary Apache APISIX ApisixRoute (replace app.example.com with your own domain):

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
 name: podinfo
 namespace: test
spec:
 http:
 - backends:
 - serviceName: podinfo
 servicePort: 80
 match:
 hosts:
 - app.example.com
 methods:
 - GET
 paths:
 - /*
 name: method
 plugins:
 - name: prometheus
 enable: true
 config:
 disable: false
 prefer_name: true

Save the above resource as podinfo-apisixroute.yaml and then apply it:

kubectl apply -f ./podinfo-apisixroute.yaml

Create a canary custom resource (replace app.example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: apisix
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # apisix route reference
 routeRef:
 apiVersion: apisix.apache.org/v2
 kind: ApisixRoute
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 80
 # container port number or name
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 10s
 # max number of failed metric checks before rollback
 threshold: 10
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 10
 # APISIX Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # builtin Prometheus check
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 type: rollout
 metadata:
 cmd: |-
 hey -z 1m -q 10 -c 2 -h2 -host app.example.com http://apisix-gateway.apisix/api/info

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
apisixroute/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
apisixroute/podinfo-podinfo-canary

Automated canary promotion

Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack or MS Teams.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Conditions:
 Message: Canary analysis completed successfully, promotion finished.
 Reason: Succeeded
 Status: True
 Type: Promoted
 Failed Checks: 1
 Iterations: 0
 Phase: Succeeded

Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Warning Synced 2m59s flagger podinfo-primary.test not ready: waiting for rollout to finish: observed deployment generation less than desired generation
 Warning Synced 2m50s flagger podinfo-primary.test not ready: waiting for rollout to finish: 0 of 1 (readyThreshold 100%) updated replicas are available
 Normal Synced 2m40s (x3 over 2m59s) flagger all the metrics providers are available!
 Normal Synced 2m39s flagger Initialization done! podinfo.test
 Normal Synced 2m20s flagger New revision detected! Scaling up podinfo.test
 Warning Synced 2m (x2 over 2m10s) flagger canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 (readyThreshold 100%) updated replicas are available
 Normal Synced 110s flagger Starting canary analysis for podinfo.test
 Normal Synced 109s flagger Advance podinfo.test canary weight 10
 Warning Synced 100s flagger Halt advancement no values found for apisix metric request-success-rate probably podinfo.test is not receiving traffic: running query failed: no values found
 Normal Synced 90s flagger Advance podinfo.test canary weight 20
 Normal Synced 80s flagger Advance podinfo.test canary weight 30
 Normal Synced 69s flagger Advance podinfo.test canary weight 40
 Normal Synced 59s flagger Advance podinfo.test canary weight 50
 Warning Synced 30s (x2 over 40s) flagger podinfo-primary.test not ready: waiting for rollout to finish: 1 old replicas are pending termination
 Normal Synced 9s (x3 over 50s) flagger (combined from similar events): Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo-2 Progressing 10 2022-11-23T05:00:54Z
test podinfo Succeeded 0 2022-11-23T06:00:54Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it deploy/flagger-loadtester bash

Generate HTTP 500 errors:

hey -z 1m -c 5 -q 5 -host app.example.com http://apisix-gateway.apisix/status/500

Generate latency:

watch -n 1 curl -H \"host: app.example.com\" http://apisix-gateway.apisix/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n apisix logs deploy/flagger -f | jq .msg

"New revision detected! Scaling up podinfo.test"
"canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 (readyThreshold 100%) updated replicas are available"
"Starting canary analysis for podinfo.test"
"Advance podinfo.test canary weight 10"
"Halt podinfo.test advancement success rate 0.00% < 99%"
"Halt podinfo.test advancement success rate 26.76% < 99%"
"Halt podinfo.test advancement success rate 34.19% < 99%"
"Halt podinfo.test advancement success rate 37.32% < 99%"
"Halt podinfo.test advancement success rate 39.04% < 99%"
"Halt podinfo.test advancement success rate 40.13% < 99%"
"Halt podinfo.test advancement success rate 48.28% < 99%"
"Halt podinfo.test advancement success rate 50.35% < 99%"
"Halt podinfo.test advancement success rate 56.92% < 99%"
"Halt podinfo.test advancement success rate 67.70% < 99%"
"Rolling back podinfo.test failed checks threshold reached 10"
"Canary failed! Scaling down podinfo.test"

Custom metrics

The canary analysis can be extended with Prometheus queries.

Create a metric template and apply it on the cluster:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: not-found-percentage
 namespace: test
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus.apisix:9090
 query: |
 sum(
 rate(
 apisix_http_status{
 route=~"{{ namespace }}_{{ route }}-{{ target }}-canary_.+",
 code!~"4.."
 }[{{ interval }}]
 )
 )
 /
 sum(
 rate(
 apisix_http_status{
 route=~"{{ namespace }}_{{ route }}-{{ target }}-canary_.+"
 }[{{ interval }}]
 )
 ) * 100

Edit the canary analysis and add the not found error rate check:

 analysis:
 metrics:
 - name: "404s percentage"
 templateRef:
 name: not-found-percentage
 thresholdRange:
 max: 5
 interval: 1m

The above configuration validates the canary by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the canary fails.

The above procedures can be extended with more custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: Linkerd Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use Linkerd and Flagger to automate canary deployments.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Linkerd 2.10 or newer.

Install Linkerd and Prometheus (part of Linkerd Viz):

# For linkerd versions 2.12 and later, the CRDs need to be installed beforehand
linkerd install --crds | kubectl apply -f -

linkerd install | kubectl apply -f -
linkerd viz install | kubectl apply -f -

# For linkerd versions 2.12 and later, the SMI extension needs to be install in
# order to enable TrafficSplits
curl -sL https://linkerd.github.io/linkerd-smi/install | sh
linkerd smi install | kubectl apply -f -

Install Flagger in the flagger-system namespace:

kubectl apply -k github.com/fluxcd/flagger//kustomize/linkerd

If you prefer Helm, these are the commands to install Linkerd, Linkerd Viz, Linkerd-SMI and Flagger:

helm repo add linkerd https://helm.linkerd.io/stable
helm install linkerd-crds linkerd/linkerd-crds -n linkerd --create-namespace
# See https://linkerd.io/2/tasks/generate-certificates/ for how to generate the
# certs referred below
helm install linkerd-control-plane linkerd/linkerd-control-plane \
 -n linkerd \
 --set-file identityTrustAnchorsPEM=ca.crt \
 --set-file identity.issuer.tls.crtPEM=issuer.crt \
 --set-file identity.issuer.tls.keyPEM=issuer.key \

helm install linkerd-viz linkerd/linkerd-viz -n linkerd-viz --create-namespace

helm repo add l5d-smi https://linkerd.github.io/linkerd-smi
helm install linkerd-smi l5d-smi/linkerd-smi -n linkerd-smi --create-namespace

# Note that linkerdAuthPolicy.create=true is only required for Linkerd 2.12 and
# later
helm install flagger flagger/flagger \
 --namespace flagger-system \
 --set meshProvider=linkerd \
 --set metricsServer=http://prometheus.linkerd-viz:9090 \
 --set linkerdAuthPolicy.create=true

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and SMI traffic split). These objects expose the application inside the mesh and drive the canary analysis and promotion.

Create a test namespace and enable Linkerd proxy injection:

kubectl create ns test
kubectl annotate namespace test linkerd.io/inject=enabled

Install the load testing service to generate traffic during the canary analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Create a canary custom resource for the podinfo deployment:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 30s
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # Linkerd Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 type: rollout
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
ingresses.extensions/podinfo
canary.flagger.app/podinfo

# generated
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
trafficsplits.split.smi-spec.io/podinfo

After the bootstrap, the podinfo deployment will be scaled to zero and the traffic to podinfo.test will be routed to the primary pods. During the canary analysis, the podinfo-canary.test address can be used to target directly the canary pods.

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

A canary deployment is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps mounted as volumes or mapped to environment variables
Secrets mounted as volumes or mapped to environment variables

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-06-30T14:05:07Z
prod frontend Succeeded 0 2019-06-30T16:15:07Z
prod backend Failed 0 2019-06-30T17:05:07Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it flagger-loadtester-xx-xx sh

Generate HTTP 500 errors:

watch -n 1 curl http://podinfo-canary.test:9898/status/500

Generate latency:

watch -n 1 curl http://podinfo-canary.test:9898/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 10
 Phase: Failed
Events:
 Starting canary analysis for podinfo.test
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Halt podinfo.test advancement success rate 69.17% < 99%
 Halt podinfo.test advancement success rate 61.39% < 99%
 Halt podinfo.test advancement success rate 55.06% < 99%
 Halt podinfo.test advancement request duration 1.20s > 0.5s
 Halt podinfo.test advancement request duration 1.45s > 0.5s
 Rolling back podinfo.test failed checks threshold reached 5
 Canary failed! Scaling down podinfo.test

Custom metrics

The canary analysis can be extended with Prometheus queries.

Let’s a define a check for not found errors. Edit the canary analysis and add the following metric:

 analysis:
 metrics:
 - name: "404s percentage"
 threshold: 3
 query: |
 100 - sum(
 rate(
 response_total{
 namespace="test",
 deployment="podinfo",
 status_code!="404",
 direction="inbound"
 }[1m]
 )
 )
 /
 sum(
 rate(
 response_total{
 namespace="test",
 deployment="podinfo",
 direction="inbound"
 }[1m]
 )
 )
 * 100

The above configuration validates the canary version by checking if the HTTP 404 req/sec percentage is below three percent of the total traffic. If the 404s rate reaches the 3% threshold, then the analysis is aborted and the canary is marked as failed.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.3

Generate 404s:

watch -n 1 curl http://podinfo-canary:9898/status/404

Watch Flagger logs:

kubectl -n linkerd logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Halt podinfo.test advancement 404s percentage 6.20 > 3
Halt podinfo.test advancement 404s percentage 6.45 > 3
Halt podinfo.test advancement 404s percentage 7.22 > 3
Halt podinfo.test advancement 404s percentage 6.50 > 3
Halt podinfo.test advancement 404s percentage 6.34 > 3
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

If you have Slack configured, Flagger will send a notification with the reason why the canary failed.

Linkerd Ingress

There are two ingress controllers that are compatible with both Flagger and Linkerd: NGINX and Gloo.

Install NGINX:

helm upgrade -i nginx-ingress stable/nginx-ingress \
--namespace ingress-nginx

Create an ingress definition for podinfo that rewrites the incoming header to the internal service name (required by Linkerd):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: podinfo
 namespace: test
 labels:
 app: podinfo
 annotations:
 kubernetes.io/ingress.class: "nginx"
 nginx.ingress.kubernetes.io/configuration-snippet: |
 proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:9898;
 proxy_hide_header l5d-remote-ip;
 proxy_hide_header l5d-server-id;
spec:
 rules:
 - host: app.example.com
 http:
 paths:
 - backend:
 serviceName: podinfo
 servicePort: 9898

When using an ingress controller, the Linkerd traffic split does not apply to incoming traffic since NGINX in running outside of the mesh. In order to run a canary analysis for a frontend app, Flagger creates a shadow ingress and sets the NGINX specific annotations.

A/B Testing

Edit podinfo canary analysis, set the provider to nginx, add the ingress reference, remove the max/step weight and add the match conditions and iterations:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # ingress reference
 provider: nginx
 ingressRef:
 apiVersion: extensions/v1beta1
 kind: Ingress
 name: podinfo
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # container port
 port: 9898
 analysis:
 interval: 1m
 threshold: 10
 iterations: 10
 match:
 # curl -H 'X-Canary: always' http://app.example.com
 - headers:
 x-canary:
 exact: "always"
 # curl -b 'canary=always' http://app.example.com
 - headers:
 cookie:
 exact: "canary"
 # Linkerd Prometheus checks
 metrics:
 - name: request-success-rate
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 thresholdRange:
 max: 500
 interval: 30s
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
 - name: load-test
 type: rollout
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 2m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com"

The above configuration will run an analysis for ten minutes targeting users that have a canary cookie set to always or those that call the service using the X-Canary: always header.

Note that the load test now targets the external address and uses the canary cookie.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.4

Flagger detects that the deployment revision changed and starts the A/B testing:

kubectl -n test describe canary/podinfo

Events:
 Starting canary deployment for podinfo.test
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary iteration 1/10
 Advance podinfo.test canary iteration 2/10
 Advance podinfo.test canary iteration 3/10
 Advance podinfo.test canary iteration 4/10
 Advance podinfo.test canary iteration 5/10
 Advance podinfo.test canary iteration 6/10
 Advance podinfo.test canary iteration 7/10
 Advance podinfo.test canary iteration 8/10
 Advance podinfo.test canary iteration 9/10
 Advance podinfo.test canary iteration 10/10
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Promotion completed! Scaling down podinfo.test

The above procedure can be extended with custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: App Mesh Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use App Mesh and Flagger to automate canary deployments. You’ll need an EKS cluster (Kubernetes >= 1.16) configured with App Mesh, you can find the installation guide here.

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services, App Mesh virtual nodes and services). These objects expose the application on the mesh and drive the canary analysis and promotion. The only App Mesh object you need to create by yourself is the mesh resource.

Create a mesh called global:

cat << EOF | kubectl apply -f -
apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
 name: global
spec:
 namespaceSelector:
 matchLabels:
 appmesh.k8s.aws/sidecarInjectorWebhook: enabled
EOF

Create a test namespace with App Mesh sidecar injection enabled:

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
 name: test
 labels:
 appmesh.k8s.aws/sidecarInjectorWebhook: enabled
EOF

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test \
--set appmesh.enabled=true \
--set "appmesh.backends[0]=podinfo" \
--set "appmesh.backends[1]=podinfo-canary"

Create a canary definition:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 annotations:
 # Enable Envoy access logging to stdout.
 appmesh.flagger.app/accesslog: enabled
 name: podinfo
 namespace: test
spec:
 # App Mesh API reference
 provider: appmesh:v1beta2
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # container port
 port: 9898
 # App Mesh ingress timeout (optional)
 timeout: 15s
 # App Mesh retry policy (optional)
 retries:
 attempts: 3
 perTryTimeout: 5s
 retryOn: "gateway-error,client-error,stream-error"
 # App Mesh URI settings
 match:
 - uri:
 prefix: /
 rewrite:
 uri: /
 # define the canary analysis timing and KPIs
 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # App Mesh Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo

# generated Kubernetes objects
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary

# generated App Mesh objects
virtualnode.appmesh.k8s.aws/podinfo-canary
virtualnode.appmesh.k8s.aws/podinfo-primary
virtualrouter.appmesh.k8s.aws/podinfo
virtualrouter.appmesh.k8s.aws/podinfo-canary
virtualservice.appmesh.k8s.aws/podinfo
virtualservice.appmesh.k8s.aws/podinfo-canary

App Mesh blocks all egress traffic by default. If your application needs to call another service, you have to create an App Mesh virtual service for it and add the virtual service name to the backend list.

 service:
 port: 9898
 backends:
 - backend1
 - arn:aws:appmesh:eu-west-1:12345678910:mesh/my-mesh/virtualService/backend2

Setup App Mesh Gateway (optional)

In order to expose the podinfo app outside the mesh you can use the App Mesh Gateway.

Deploy the App Mesh Gateway behind an AWS NLB:

helm upgrade -i appmesh-gateway eks/appmesh-gateway \
--namespace test

Find the gateway public address:

export URL="http://$(kubectl -n test get svc/appmesh-gateway -ojson | jq -r ".status.loadBalancer.ingress[].hostname")"
echo $URL

Wait for the NLB to become active:

 watch curl -sS $URL

Create a gateway route that points to the podinfo virtual service:

cat << EOF | kubectl apply -f -
apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
metadata:
 name: podinfo
 namespace: test
spec:
 httpRoute:
 match:
 prefix: "/"
 action:
 target:
 virtualService:
 virtualServiceRef:
 name: podinfo
EOF

Open your browser and navigate to the ingress address to access podinfo UI.

Automated canary promotion

A canary deployment is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps and Secrets mounted as volumes or mapped to environment variables

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Routing all traffic to primary
 Promotion completed! Scaling down podinfo.test

When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

During the analysis the canary’s progress can be monitored with Grafana. The App Mesh dashboard URL is http://localhost:3000/d/flagger-appmesh/appmesh-canary?refresh=10s&orgId=1&var-namespace=test&var-primary=podinfo-primary&var-canary=podinfo.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT
test podinfo Progressing 15
prod frontend Succeeded 0
prod backend Failed 0

If you’ve enabled the Slack notifications, you should receive the following messages:

Automated rollback

During the canary analysis you can generate HTTP 500 errors or high latency to test if Flagger pauses the rollout.

Trigger a canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it deploy/flagger-loadtester bash

Generate HTTP 500 errors:

hey -z 1m -c 5 -q 5 http://podinfo-canary.test:9898/status/500

Generate latency:

watch -n 1 curl http://podinfo-canary.test:9898/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n appmesh-system logs deploy/flagger -f | jq .msg

New revision detected! progressing canary analysis for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement request duration 1.20s > 0.5s
Halt podinfo.test advancement request duration 1.45s > 0.5s
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

If you’ve enabled the Slack notifications, you’ll receive a message if the progress deadline is exceeded, or if the analysis reached the maximum number of failed checks:

A/B Testing

Edit the canary analysis, remove the max/step weight and add the match conditions and iterations:

 analysis:
 interval: 1m
 threshold: 5
 iterations: 10
 match:
 - headers:
 x-canary:
 exact: "insider"
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 -H 'X-Canary: insider' http://podinfo.test:9898/"

The above configuration will run an analysis for ten minutes targeting users that have a X-Canary: insider header.

You can also use a HTTP cookie, to target all users with a canary cookie set to insider the match condition should be:

match:
- headers:
 cookie:
 regex: "^(.*?;)?(canary=insider)(;.*)?$"
webhooks:
- name: load-test
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 -H 'Cookie: canary=insider' http://podinfo.test:9898/"

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.3

Flagger detects that the deployment revision changed and starts the A/B test:

kubectl -n appmesh-system logs deploy/flagger -f | jq .msg

New revision detected! progressing canary analysis for podinfo.test
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Routing all traffic to primary
Promotion completed! Scaling down podinfo.test

The above procedure can be extended with custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: Contour Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use Contour ingress controller and Flagger to automate canary releases and A/B testing.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Contour v1.0 or newer.

Install Contour on a cluster with LoadBalancer support:

kubectl apply -f https://projectcontour.io/quickstart/contour.yaml

The above command will deploy Contour and an Envoy daemonset in the projectcontour namespace.

Install Flagger using Kustomize (kubectl 1.14) in the projectcontour namespace:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/contour?ref=main

The above command will deploy Flagger and Prometheus configured to scrape the Contour’s Envoy instances.

Or you can install Flagger using Helm v3:

helm repo add flagger https://flagger.app

helm upgrade -i flagger flagger/flagger \
--namespace projectcontour \
--set meshProvider=contour \
--set ingressClass=contour \
--set prometheus.install=true

You can also enable Slack, Discord, Rocket or MS Teams notifications, see the alerting docs.

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and Contour HTTPProxy). These objects expose the application in the cluster and drive the canary analysis and promotion.

Create a test namespace:

kubectl create ns test

Install the load testing service to generate traffic during the canary analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Create a canary custom resource (replace app.example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # HPA reference
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # service port
 port: 80
 # container port
 targetPort: 9898
 # Contour request timeout
 timeout: 15s
 # Contour retry policy
 retries:
 attempts: 3
 perTryTimeout: 5s
 # supported values for retryOn - https://projectcontour.io/docs/main/config/api/#projectcontour.io/v1.RetryOn
 retryOn: "5xx"
 # define the canary analysis timing and KPIs
 analysis:
 # schedule interval (default 60s)
 interval: 30s
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # Contour Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99 in milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary.test/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 type: rollout
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 -host app.example.com http://envoy.projectcontour"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

The canary analysis will run for five minutes while validating the HTTP metrics and rollout hooks every half a minute.

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo

# generated
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
httpproxy.projectcontour.io/podinfo

Expose the app outside the cluster

Find the external address of Contour’s Envoy load balancer:

export ADDRESS="$(kubectl -n projectcontour get svc/envoy -ojson \
| jq -r ".status.loadBalancer.ingress[].hostname")"
echo $ADDRESS

Configure your DNS server with a CNAME record (AWS) or A record (GKE/AKS/DOKS) and point a domain e.g. app.example.com to the LB address.

Create a HTTPProxy definition and include the podinfo proxy generated by Flagger (replace app.example.com with your own domain):

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
 name: podinfo-ingress
 namespace: test
spec:
 virtualhost:
 fqdn: app.example.com
 includes:
 - name: podinfo
 namespace: test
 conditions:
 - prefix: /

Save the above resource as podinfo-ingress.yaml and then apply it:

kubectl apply -f ./podinfo-ingress.yaml

Verify that Contour processed the proxy definition with:

kubectl -n test get httpproxies

NAME FQDN STATUS
podinfo valid
podinfo-ingress app.example.com valid

Now you can access podinfo UI using your domain address.

Note that you should be using HTTPS when exposing production workloads on internet. You can obtain free TLS certs from Let’s Encrypt, read this guide on how to configure cert-manager to secure Contour with TLS certificates.

Automated canary promotion

A canary deployment is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps and Secrets mounted as volumes or mapped to environment variables

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Routing all traffic to primary
 Promotion completed! Scaling down podinfo.test

When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary.

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-12-20T14:05:07Z

If you’ve enabled the Slack notifications, you should receive the following messages:

Automated rollback

During the canary analysis you can generate HTTP 500 errors or high latency to test if Flagger pauses the rollout.

Trigger a canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it deploy/flagger-loadtester bash

Generate HTTP 500 errors:

hey -z 1m -c 5 -q 5 http://app.example.com/status/500

Generate latency:

watch -n 1 curl http://app.example.com/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n projectcontour logs deploy/flagger -f | jq .msg

New revision detected! progressing canary analysis for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement success rate 69.17% < 99%
Halt podinfo.test advancement success rate 61.39% < 99%
Halt podinfo.test advancement success rate 55.06% < 99%
Halt podinfo.test advancement request duration 1.20s > 500ms
Halt podinfo.test advancement request duration 1.45s > 500ms
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

If you’ve enabled the Slack notifications, you’ll receive a message if the progress deadline is exceeded, or if the analysis reached the maximum number of failed checks:

A/B Testing

Edit the canary analysis, remove the max/step weight and add the match conditions and iterations:

analysis:
 interval: 1m
 threshold: 5
 iterations: 10
 match:
 - headers:
 x-canary:
 exact: "insider"
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 1m -q 5 -c 5 -H 'X-Canary: insider' -host app.example.com http://envoy.projectcontour"

The above configuration will run an analysis for ten minutes targeting users that have a X-Canary: insider header.

You can also use a HTTP cookie. To target all users with a cookie set to insider, the match condition should be:

match:
- headers:
 cookie:
 suffix: "insider"
webhooks:
- name: load-test
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 1m -q 5 -c 5 -H 'Cookie: canary=insider' -host app.example.com http://envoy.projectcontour"

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.3

Flagger detects that the deployment revision changed and starts the A/B test:

kubectl -n projectcontour logs deploy/flagger -f | jq .msg

New revision detected! Progressing canary analysis for podinfo.test
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Routing all traffic to primary
Promotion completed! Scaling down podinfo.test

The web browser user agent header allows user segmentation based on device or OS.

For example, if you want to route all mobile users to the canary instance:

match:
- headers:
 user-agent:
 prefix: "Mobile"

Or if you want to target only Android users:

match:
- headers:
 user-agent:
 prefix: "Android"

Or a specific browser version:

match:
- headers:
 user-agent:
 suffix: "Firefox/71.0"

For an in-depth look at the analysis process read the usage docs.

Flagger: Gloo Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use the Gloo Edge ingress controller and Flagger to automate canary releases and A/B testing.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Gloo Edge ingress 1.6.0 or newer.

This guide was written for Flagger version 1.6.0 or higher. Prior versions of Flagger used Gloo UpstreamGroups to handle canaries, but newer versions of Flagger use Gloo RouteTables to handle canaries as well as A/B testing.

Install Gloo with Helm v3:

helm repo add gloo https://storage.googleapis.com/solo-public-helm
kubectl create ns gloo-system
helm upgrade -i gloo gloo/gloo \
--namespace gloo-system

Install Flagger and the Prometheus add-on in the same namespace as Gloo:

helm repo add flagger https://flagger.app

helm upgrade -i flagger flagger/flagger \
--namespace gloo-system \
--set prometheus.install=true \
--set meshProvider=gloo

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services, Gloo route tables and upstreams). These objects expose the application outside the cluster and drive the canary analysis and promotion.

Create a test namespace:

kubectl create ns test

Create a deployment and a horizontal pod autoscaler:

kubectl -n test apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

kubectl -n test apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a virtual service definition that references a route table that will be generated by Flagger (replace app.example.com with your own domain):

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
 name: podinfo
 namespace: test
spec:
 virtualHost:
 domains:
 - 'app.example.com'
 routes:
 - matchers:
 - prefix: /
 delegateAction:
 ref:
 name: podinfo
 namespace: test

Save the above resource as podinfo-virtualservice.yaml and then apply it:

kubectl apply -f ./podinfo-virtualservice.yaml

Create a canary custom resource (replace app.example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # upstreamRef (optional)
 # defines an upstream to copy the spec from when flagger generates new upstreams.
 # necessary to copy over TLS config, circuit breakers, etc. (anything nonstandard)
# upstreamRef:
# apiVersion: gloo.solo.io/v1
# kind: Upstream
# name: podinfo-upstream
# namespace: gloo-system
 provider: gloo
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # ClusterIP port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 10s
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # Gloo Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 10s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 2m -q 5 -c 2 -host app.example.com http://gateway-proxy.gloo-system"

Note: when using upstreamRef the following fields are copied over from the original upstream: Labels, SslConfig, CircuitBreakers, ConnectionConfig, UseHttp2, InitialStreamWindowSize

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
virtualservices.gateway.solo.io/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
routetables.gateway.solo.io/podinfo
upstreams.gloo.solo.io/test-podinfo-canaryupstream-9898
upstreams.gloo.solo.io/test-podinfo-primaryupstream-9898

When the bootstrap finishes Flagger will set the canary status to initialized:

kubectl -n test get canary podinfo

NAME STATUS WEIGHT LASTTRANSITIONTIME
podinfo Initialized 0 2019-05-17T08:09:51Z

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Scaling up podinfo.test
 Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 2m flagger Advance podinfo.test canary weight 20
 Normal Synced 2m flagger Advance podinfo.test canary weight 25
 Normal Synced 1m flagger Advance podinfo.test canary weight 30
 Normal Synced 1m flagger Advance podinfo.test canary weight 35
 Normal Synced 55s flagger Advance podinfo.test canary weight 40
 Normal Synced 45s flagger Advance podinfo.test canary weight 45
 Normal Synced 35s flagger Advance podinfo.test canary weight 50
 Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
 Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-05-17T14:05:07Z
prod frontend Succeeded 0 2019-05-17T16:15:07Z
prod backend Failed 0 2019-05-17T17:05:07Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Generate HTTP 500 errors:

watch curl -H 'Host: app.example.com' http://gateway-proxy.gloo-system/status/500

Generate high latency:

watch curl -H 'Host: app.example.com' http://gateway-proxy.gloo-system/delay/2

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 10
 Phase: Failed
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger Starting canary deployment for podinfo.test
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 55.06% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 47.00% < 99%
 Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement success rate 38.08% < 99%
 Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
 Warning Synced 1m flagger Canary failed! Scaling down podinfo.test

Custom metrics

The canary analysis can be extended with Prometheus queries.

The demo app is instrumented with Prometheus so you can create a custom check that will use the HTTP request duration histogram to validate the canary.

Create a metric template and apply it on the cluster:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: not-found-percentage
 namespace: test
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus.gloo-system:9090
 query: |
 100 - sum(
 rate(
 http_request_duration_seconds_count{
 kubernetes_namespace="{{ namespace }}",
 kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
 status!="{{ interval }}"
 }[1m]
 )
 )
 /
 sum(
 rate(
 http_request_duration_seconds_count{
 kubernetes_namespace="{{ namespace }}",
 kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
 }[{{ interval }}]
 )
 ) * 100

Edit the canary analysis and add the following metric:

 analysis:
 metrics:
 - name: "404s percentage"
 templateRef:
 name: not-found-percentage
 thresholdRange:
 max: 5
 interval: 1m

The above configuration validates the canary by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the canary fails.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.3

Generate 404s:

watch curl -H 'Host: app.example.com' http://gateway-proxy.gloo-system/status/404

Watch Flagger logs:

kubectl -n gloo-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement 404s percentage 6.20 > 5
Halt podinfo.test advancement 404s percentage 6.45 > 5
Halt podinfo.test advancement 404s percentage 7.60 > 5
Halt podinfo.test advancement 404s percentage 8.69 > 5
Halt podinfo.test advancement 404s percentage 9.70 > 5
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

If you have alerting configured, Flagger will send a notification with the reason why the canary failed.

A/B Testing

Edit the canary analysis, remove the max/step weight and add the match conditions and iterations:

analysis:
 interval: 1m
 threshold: 5
 iterations: 10
 match:
 - headers:
 x-canary:
 exact: "insider"
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 1m -q 5 -c 5 -H 'X-Canary: insider' -host app.example.com http://gateway-proxy.gloo-system"

The above configuration will run an analysis for ten minutes targeting users that have a X-Canary: insider header.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.4

Flagger detects that the deployment revision changed and starts the A/B test:

kubectl -n gloo-system logs deploy/flagger -f | jq .msg

New revision detected! Progressing canary analysis for podinfo.test
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Routing all traffic to primary
Promotion completed! Scaling down podinfo.test

The web browser user agent header allows user segmentation based on device or OS.

For example, if you want to route all mobile users to the canary instance:

match:
- headers:
 user-agent:
 regex: ".*Mobile.*"

Or if you want to target only Android users:

match:
- headers:
 user-agent:
 regex: ".*Android.*"

Or a specific browser version:

match:
- headers:
 user-agent:
 regex: ".*Firefox.*"

For an in-depth look at the analysis process read the usage docs.

Flagger: NGINX Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use the NGINX ingress controller and Flagger to automate canary deployments and A/B testing.

Prerequisites

Flagger requires a Kubernetes cluster v1.19 or newer and NGINX ingress v1.0.2 or newer.

Install the NGINX ingress controller with Helm v3:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
kubectl create ns ingress-nginx
helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--set controller.metrics.enabled=true \
--set controller.podAnnotations."prometheus\.io/scrape"=true \
--set controller.podAnnotations."prometheus\.io/port"=10254

Install Flagger and the Prometheus add-on in the same namespace as the ingress controller:

helm repo add flagger https://flagger.app

helm upgrade -i flagger flagger/flagger \
--namespace ingress-nginx \
--set prometheus.install=true \
--set meshProvider=nginx

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and canary ingress). These objects expose the application outside the cluster and drive the canary analysis and promotion.

Create a test namespace:

kubectl create ns test

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test

Create an ingress definition (replace app.example.com with your own domain):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: podinfo
 namespace: test
 labels:
 app: podinfo
 annotations:
 kubernetes.io/ingress.class: "nginx"
spec:
 rules:
 - host: "app.example.com"
 http:
 paths:
 - pathType: Prefix
 path: "/"
 backend:
 service:
 name: podinfo
 port:
 number: 80

Save the above resource as podinfo-ingress.yaml and then apply it:

kubectl apply -f ./podinfo-ingress.yaml

Create a canary custom resource (replace app.example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: nginx
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # ingress reference
 ingressRef:
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 name: podinfo
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 80
 # container port number or name
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 10s
 # max number of failed metric checks before rollback
 threshold: 10
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # NGINX Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 http://app.example.com/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
ingresses.extensions/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
ingresses.extensions/podinfo-canary

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Scaling up podinfo.test
 Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 2m flagger Advance podinfo.test canary weight 20
 Normal Synced 2m flagger Advance podinfo.test canary weight 25
 Normal Synced 1m flagger Advance podinfo.test canary weight 30
 Normal Synced 1m flagger Advance podinfo.test canary weight 35
 Normal Synced 55s flagger Advance podinfo.test canary weight 40
 Normal Synced 45s flagger Advance podinfo.test canary weight 45
 Normal Synced 35s flagger Advance podinfo.test canary weight 50
 Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
 Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-05-06T14:05:07Z
prod frontend Succeeded 0 2019-05-05T16:15:07Z
prod backend Failed 0 2019-05-04T17:05:07Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Generate HTTP 500 errors:

watch curl http://app.example.com/status/500

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 10
 Phase: Failed
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger Starting canary deployment for podinfo.test
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 55.06% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 47.00% < 99%
 Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement success rate 38.08% < 99%
 Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
 Warning Synced 1m flagger Canary failed! Scaling down podinfo.test

Custom metrics

The canary analysis can be extended with Prometheus queries.

The demo app is instrumented with Prometheus so you can create a custom check that will use the HTTP request duration histogram to validate the canary.

Create a metric template and apply it on the cluster:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: latency
 namespace: test
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus.ingress-nginx:9090
 query: |
 histogram_quantile(0.99,
 sum(
 rate(
 http_request_duration_seconds_bucket{
 kubernetes_namespace="{{ namespace }}",
 kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
 }[1m]
 )
 ) by (le)
 )

Edit the canary analysis and add the latency check:

 analysis:
 metrics:
 - name: "latency"
 templateRef:
 name: latency
 thresholdRange:
 max: 0.5
 interval: 1m

The threshold is set to 500ms so if the average request duration in the last minute goes over half a second then the analysis will fail and the canary will not be promoted.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.3

Generate high response latency:

watch curl http://app.example.com/delay/2

Watch Flagger logs:

kubectl -n nginx-ingress logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement latency 1.20 > 0.5
Halt podinfo.test advancement latency 1.45 > 0.5
Halt podinfo.test advancement latency 1.60 > 0.5
Halt podinfo.test advancement latency 1.69 > 0.5
Halt podinfo.test advancement latency 1.70 > 0.5
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

If you have alerting configured, Flagger will send a notification with the reason why the canary failed.

A/B Testing

Edit the canary analysis, remove the max/step weight and add the match conditions and iterations:

 analysis:
 interval: 1m
 threshold: 10
 iterations: 10
 match:
 # curl -H 'X-Canary: insider' http://app.example.com
 - headers:
 x-canary:
 exact: "insider"
 # curl -b 'canary=always' http://app.example.com
 - headers:
 cookie:
 exact: "canary"
 metrics:
 - name: request-success-rate
 thresholdRange:
 min: 99
 interval: 1m
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 1m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com/"

The above configuration will run an analysis for ten minutes targeting users that have a canary cookie set to always or those that call the service using the X-Canary: insider header.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.4

Flagger detects that the deployment revision changed and starts the A/B testing:

kubectl -n test describe canary/podinfo

Status:
 Failed Checks: 0
 Phase: Succeeded
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Scaling up podinfo.test
 Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
 Normal Synced 2m flagger Advance podinfo.test canary iteration 4/10
 Normal Synced 2m flagger Advance podinfo.test canary iteration 5/10
 Normal Synced 1m flagger Advance podinfo.test canary iteration 6/10
 Normal Synced 1m flagger Advance podinfo.test canary iteration 7/10
 Normal Synced 55s flagger Advance podinfo.test canary iteration 8/10
 Normal Synced 45s flagger Advance podinfo.test canary iteration 9/10
 Normal Synced 35s flagger Advance podinfo.test canary iteration 10/10
 Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
 Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test

The above procedure can be extended with custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: Skipper Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use the Skipper ingress controller and Flagger to automate canary deployments.

Prerequisites

Flagger requires a Kubernetes cluster v1.19 or newer and Skipper ingress v0.13 or newer.

Install Skipper ingress-controller using upstream definition.

Certain arguments are relevant:

- -enable-connection-metrics
- -histogram-metric-buckets=.01,1,10,100
- -kubernetes
- -kubernetes-in-cluster
- -kubernetes-path-mode=path-prefix
- -metrics-exp-decay-sample
- -metrics-flavour=prometheus
- -route-backend-metrics
- -route-backend-error-counters
- -route-response-metrics
- -serve-host-metrics
- -serve-route-metrics
- -whitelisted-healthcheck-cidr=0.0.0.0/0 # permit Kind source health checks

Install Flagger using kustomize:

kustomize build https://github.com/fluxcd/flagger/kustomize/kubernetes | kubectl apply -f -

Bootstrap

Create a test namespace:

kubectl create ns test

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test

Create an ingress definition (replace app.example.com with your own domain):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: podinfo
 namespace: test
 labels:
 app: podinfo
 annotations:
 kubernetes.io/ingress.class: "skipper"
spec:
 rules:
 - host: "app.example.com"
 http:
 paths:
 - pathType: Prefix
 path: "/"
 backend:
 service:
 name: podinfo
 port:
 number: 80

Save the above resource as podinfo-ingress.yaml and then apply it:

kubectl apply -f ./podinfo-ingress.yaml

Create a canary custom resource (replace app.example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: skipper
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # ingress reference
 ingressRef:
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 name: podinfo
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 80
 # container port number or name
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 10s
 # max number of failed metric checks before rollback
 threshold: 10
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # Skipper Prometheus checks
 metrics:
 - name: request-success-rate
 interval: 1m
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 - name: request-duration
 interval: 1m
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 webhooks:
 - name: gate
 type: confirm-rollout
 url: http://flagger-loadtester.test/gate/approve
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 10s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary/token | grep token"
 - name: load-test
 type: rollout
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 10m -q 10 -c 2 -host app.example.com http://skipper-ingress.kube-system"
 logCmdOutput: "true"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
ingress.networking.k8s.io/podinfo-ingress
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
ingress.networking.k8s.io/podinfo-canary

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:4.0.6

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Routing all traffic to primary
 Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo-2 Progressing 30 2020-08-14T12:32:12Z
test podinfo Succeeded 0 2020-08-14T11:23:88Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:4.0.6

Exec into the load tester pod with:

kubectl -n test exec -it deploy/flagger-loadtester bash

Generate HTTP 500 errors:

hey -z 1m -c 5 -q 5 http://app.example.com/status/500

Generate latency:

watch -n 1 curl http://app.example.com/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n flagger-system logs deploy/flagger -f | jq .msg

New revision detected! Scaling up podinfo.test
Canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 updated replicas are available
Starting canary analysis for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Halt podinfo.test advancement success rate 53.42% < 99%
Halt podinfo.test advancement success rate 53.19% < 99%
Halt podinfo.test advancement success rate 48.05% < 99%
Rolling back podinfo.test failed checks threshold reached 3
Canary failed! Scaling down podinfo.test

Custom metrics

The canary analysis can be extended with Prometheus queries.

Create a metric template and apply it on the cluster:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: latency
 namespace: test
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus.flagger-system:9090
 query: |
 histogram_quantile(0.99,
 sum(
 rate(
 skipper_serve_route_duration_seconds_bucket{
 route=~"{{ printf "kube(ew)?_%s__%s_canary__.*__%s_canary(_[0-9]+)?" namespace ingress service }}",
 le="+Inf"
 }[1m]
 )
 ) by (le)
 )

Edit the canary analysis and add the latency check:

 analysis:
 metrics:
 - name: "latency"
 templateRef:
 name: latency
 thresholdRange:
 max: 0.5
 interval: 1m

The threshold is set to 500ms so if the average request duration in the last minute goes over half a second then the analysis will fail and the canary will not be promoted.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:4.0.6

Generate high response latency:

watch curl http://app.example.com/delay/2

Watch Flagger logs:

kubectl -n flagger-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement latency 1.20 > 0.5
Halt podinfo.test advancement latency 1.45 > 0.5
Halt podinfo.test advancement latency 1.60 > 0.5
Halt podinfo.test advancement latency 1.69 > 0.5
Halt podinfo.test advancement latency 1.70 > 0.5
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

If you have alerting configured, Flagger will send a notification with the reason why the canary failed.

Flagger: Traefik Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use the Traefik and Flagger to automate canary deployments.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Traefik v2.3 or newer.

Install Traefik with Helm v3:

helm repo add traefik https://helm.traefik.io/traefik
kubectl create ns traefik

cat <<EOF | helm upgrade -i traefik traefik/traefik --namespace traefik -f -
deployment:
 podAnnotations:
 prometheus.io/port: "9100"
 prometheus.io/scrape: "true"
 prometheus.io/path: "/metrics"
metrics:
 prometheus:
 entryPoint: metrics
EOF

Install Flagger and the Prometheus add-on in the same namespace as Traefik:

helm repo add flagger https://flagger.app

helm upgrade -i flagger flagger/flagger \
--namespace traefik \
--set prometheus.install=true \
--set meshProvider=traefik

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and TraefikService). These objects expose the application outside the cluster and drive the canary analysis and promotion.

Create a test namespace:

kubectl create ns test

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

helm upgrade -i flagger-loadtester flagger/loadtester \
--namespace=test

Create Traefik IngressRoute that references TraefikService generated by Flagger (replace app.example.com with your own domain):

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
 name: podinfo
 namespace: test
spec:
 entryPoints:
 - web
 routes:
 - match: Host(`app.example.com`)
 kind: Rule
 services:
 - name: podinfo
 kind: TraefikService
 port: 80

Save the above resource as podinfo-ingressroute.yaml and then apply it:

kubectl apply -f ./podinfo-ingressroute.yaml

Create a canary custom resource (replace app.example.com with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: traefik
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 80
 # container port number or name
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 10s
 # max number of failed metric checks before rollback
 threshold: 10
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # Traefik Prometheus checks
 metrics:
 - name: request-success-rate
 interval: 1m
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 - name: request-duration
 interval: 1m
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 10s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary.test/token | grep token"
 - name: load-test
 type: rollout
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 10m -q 10 -c 2 -host app.example.com http://traefik.traefik"
 logCmdOutput: "true"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
traefikservice.traefik.containo.us/podinfo

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:4.0.6

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Routing all traffic to primary
 Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo-2 Progressing 30 2020-08-14T12:32:12Z
test podinfo Succeeded 0 2020-08-14T11:23:88Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:4.0.6

Exec into the load tester pod with:

kubectl -n test exec -it deploy/flagger-loadtester bash

Generate HTTP 500 errors:

hey -z 1m -c 5 -q 5 http://app.example.com/status/500

Generate latency:

watch -n 1 curl http://app.example.com/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n traefik logs deploy/flagger -f | jq .msg

New revision detected! Scaling up podinfo.test
Canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 updated replicas are available
Starting canary analysis for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Halt podinfo.test advancement success rate 53.42% < 99%
Halt podinfo.test advancement success rate 53.19% < 99%
Halt podinfo.test advancement success rate 48.05% < 99%
Rolling back podinfo.test failed checks threshold reached 3
Canary failed! Scaling down podinfo.test

Custom metrics

The canary analysis can be extended with Prometheus queries.

Create a metric template and apply it on the cluster:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: not-found-percentage
 namespace: test
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus.traefik:9090
 query: |
 sum(
 rate(
 traefik_service_request_duration_seconds_bucket{
 service=~"{{ namespace }}-{{ target }}-canary-[0-9a-zA-Z-]+@kubernetescrd",
 code!="404",
 }[{{ interval }}]
 )
 )
 /
 sum(
 rate(
 traefik_service_request_duration_seconds_bucket{
 service=~"{{ namespace }}-{{ target }}-canary-[0-9a-zA-Z-]+@kubernetescrd",
 }[{{ interval }}]
 )
 ) * 100

Edit the canary analysis and add the not found error rate check:

 analysis:
 metrics:
 - name: "404s percentage"
 templateRef:
 name: not-found-percentage
 thresholdRange:
 max: 5
 interval: 1m

The above configuration validates the canary by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the canary fails.

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:4.0.6

Generate 404s:

watch curl http://app.example.com/status/400

Watch Flagger logs:

kubectl -n traefik logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement 404s percentage 6.20 > 5
Halt podinfo.test advancement 404s percentage 6.45 > 5
Halt podinfo.test advancement 404s percentage 7.60 > 5
Halt podinfo.test advancement 404s percentage 8.69 > 5
Halt podinfo.test advancement 404s percentage 9.70 > 5
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

If you have alerting configured, Flagger will send a notification with the reason why the canary failed.

For an in-depth look at the analysis process read the usage docs.

Flagger: Open Service Mesh Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use Open Service Mesh (OSM) and Flagger to automate canary deployments.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Open Service Mesh 0.9.1 or newer.

OSM must have permissive traffic policy enabled and have an instance of Prometheus for metrics.

If the OSM CLI is being used for installation, install OSM using the following command:

osm install \
--set=OpenServiceMesh.deployPrometheus=true \
--set=OpenServiceMesh.enablePermissiveTrafficPolicy=true

If a managed instance of OSM is being used:
- Bring your own instance of Prometheus, setting the namespace to match the managed OSM controller namespace
- Enable permissive traffic policy after installation by updating the OSM MeshConfig resource:
```
# Replace <osm-namespace> with OSM controller's namespace
kubectl patch meshconfig osm-mesh-config -n <osm-namespace> -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":true}}}' --type=merge
```

To install Flagger in the default osm-system namespace, use:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/osm?ref=main

Alternatively, if a non-default namespace or managed instance of OSM is in use, install Flagger with Helm, replacing the values as appropriate. If a custom instance of Prometheus is being used, replace osm-prometheus with the relevant Prometheus service name.

helm upgrade -i flagger flagger/flagger \
--namespace=<osm-namespace> \
--set meshProvider=osm \
--set metricsServer=http://osm-prometheus.<osm-namespace>.svc:7070

Bootstrap

Create a test namespace and enable OSM namespace monitoring and metrics scraping for the namespace.

kubectl create namespace test
osm namespace add test
osm metrics enable --namespace test

Create a podinfo deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Install the load testing service to generate traffic during the canary analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a canary custom resource for the podinfo deployment. The following podinfo canary custom resource instructs Flagger to:

monitor any changes to the podinfo deployment created earlier,
detect podinfo deployment revision changes, and
start a Flagger canary analysis, rollout, and promotion if there were deployment revision changes.

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: osm
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rolled back (default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 analysis:
 # schedule interval (default 60s)
 interval: 30s
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 # OSM Prometheus checks
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # testing (optional)
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 type: rollout
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects.

# applied
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
ingresses.extensions/podinfo
canary.flagger.app/podinfo

# generated
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
trafficsplits.split.smi-spec.io/podinfo

After the bootstrap, the podinfo deployment will be scaled to zero and the traffic to podinfo.test will be routed to the primary pods. During the canary analysis, the podinfo-canary.test address can be used to target directly the canary pods.

Automated Canary Promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Promotion completed! Scaling down podinfo.test

Note that if you apply any new changes to the podinfo deployment during the canary analysis, Flagger will restart the analysis.

A canary deployment is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps mounted as volumes or mapped to environment variables
Secrets mounted as volumes or mapped to environment variables

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-06-30T14:05:07Z
prod frontend Succeeded 0 2019-06-30T16:15:07Z
prod backend Failed 0 2019-06-30T17:05:07Z

Automated Rollback

During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it flagger-loadtester-xx-xx sh

Repeatedly generate HTTP 500 errors until the kubectl describe output below shows canary rollout failure:

watch -n 0.1 curl http://podinfo-canary.test:9898/status/500

Repeatedly generate latency until canary rollout fails:

watch -n 0.1 curl http://podinfo-canary.test:9898/delay/1

When the number of failed checks reaches the canary analysis thresholds defined in the podinfo canary custom resource earlier, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 10
 Phase: Failed
Events:
 Starting canary analysis for podinfo.test
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Halt podinfo.test advancement success rate 69.17% < 99%
 Halt podinfo.test advancement success rate 61.39% < 99%
 Halt podinfo.test advancement success rate 55.06% < 99%
 Halt podinfo.test advancement request duration 1.20s > 0.5s
 Halt podinfo.test advancement request duration 1.45s > 0.5s
 Rolling back podinfo.test failed checks threshold reached 5
 Canary failed! Scaling down podinfo.test

Custom Metrics

The canary analysis can be extended with Prometheus queries.

Let’s define a check for 404 not found errors. Edit the canary analysis (podinfo-canary.yaml file) and add the following metric. For more information on creating additional custom metrics using OSM metrics, please check the metrics available in OSM.

 analysis:
 metrics:
 - name: "404s percentage"
 threshold: 3
 query: |
 100 - (
 sum(
 rate(
 osm_request_total{
 destination_namespace="test",
 destination_kind="Deployment",
 destination_name="podinfo",
 response_code!="404"
 }[1m]
 )
 )
 /
 sum(
 rate(
 osm_request_total{
 destination_namespace="test",
 destination_kind="Deployment",
 destination_name="podinfo"
 }[1m]
 )
 ) * 100
 )

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.3

Exec into the load tester pod with:

kubectl -n test exec -it flagger-loadtester-xx-xx sh

Repeatedly generate 404s until canary rollout fails:

watch -n 0.1 curl http://podinfo-canary.test:9898/status/404

Watch Flagger logs to confirm successful canary rollback.

kubectl -n osm-system logs deployment/flagger -f | jq .msg

Starting canary deployment for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Halt podinfo.test advancement 404s percentage 6.20 > 3
Halt podinfo.test advancement 404s percentage 6.45 > 3
Halt podinfo.test advancement 404s percentage 7.22 > 3
Halt podinfo.test advancement 404s percentage 6.50 > 3
Halt podinfo.test advancement 404s percentage 6.34 > 3
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test

Flagger: Kuma Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use Kuma and Flagger to automate canary deployments.

Prerequisites

Flagger requires a Kubernetes cluster v1.19 or newer and Kuma 1.7 or newer.

Install Kuma and Prometheus (part of Kuma Metrics):

kumactl install control-plane | kubectl apply -f -
kumactl install observability --components "grafana,prometheus" | kubectl apply -f -

Install Flagger in the kuma-system namespace:

kubectl apply -k github.com/fluxcd/flagger//kustomize/kuma

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and Kuma TrafficRoute). These objects expose the application inside the mesh and drive the canary analysis and promotion.

Create a test namespace and enable Kuma sidecar injection:

kubectl create ns test
kubectl annotate namespace test kuma.io/sidecar-injection=enabled

Install the load testing service to generate traffic during the canary analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Create a canary custom resource for the podinfo deployment:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
 annotations:
 kuma.io/mesh: default
spec:
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 progressDeadlineSeconds: 60
 service:
 port: 9898
 targetPort: 9898
 apex:
 annotations:
 9898.service.kuma.io/protocol: "http"
 canary:
 annotations:
 9898.service.kuma.io/protocol: "http"
 primary:
 annotations:
 9898.service.kuma.io/protocol: "http"
 analysis:
 # schedule interval (default 60s)
 interval: 30s
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
 metrics:
 - name: request-success-rate
 threshold: 99
 interval: 1m
 - name: request-duration
 threshold: 500
 interval: 30s
 webhooks:
 - name: acceptance-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
 cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 type: rollout
 url: http://flagger-loadtester.test/
 metadata:
 cmd: "hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
ingresses.extensions/podinfo
canary.flagger.app/podinfo

# generated
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
trafficroutes.kuma.io/podinfo

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo-primary.test
 Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

A canary deployment is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps mounted as volumes or mapped to environment variables
Secrets mounted as volumes or mapped to environment variables

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2019-06-30T14:05:07Z
prod frontend Succeeded 0 2019-06-30T16:15:07Z
prod backend Failed 0 2019-06-30T17:05:07Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses and rolls back the faulted version.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it flagger-loadtester-xx-xx sh

Generate HTTP 500 errors:

watch -n 1 curl http://podinfo-canary.test:9898/status/500

Generate latency:

watch -n 1 curl http://podinfo-canary.test:9898/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 10
 Phase: Failed
Events:
 Starting canary analysis for podinfo.test
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Halt podinfo.test advancement success rate 69.17% < 99%
 Halt podinfo.test advancement success rate 61.39% < 99%
 Halt podinfo.test advancement success rate 55.06% < 99%
 Halt podinfo.test advancement request duration 1.20s > 0.5s
 Halt podinfo.test advancement request duration 1.45s > 0.5s
 Rolling back podinfo.test failed checks threshold reached 5
 Canary failed! Scaling down podinfo.test

The above procedures can be extended with custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: Gateway API Canary Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use Gateway API and Flagger to automate canary deployments and A/B testing.

Prerequisites

Flagger requires a Kubernetes cluster v1.19 or newer and any mesh/ingress that implements the v1beta1 version of Gateway API. We’ll be using Contour for the sake of this tutorial, but you can use any other implementation.

Note: Flagger supports v1alpha2 version of Gateway API, but the alpha version has been deprecated and support will be dropped in a future release.

Install Contour, its Gateway provisioner and Gateway API CRDs in the projectcontour namespace:

https://raw.githubusercontent.com/projectcontour/contour/release-1.23/examples/render/contour-gateway-provisioner.yaml

Alternatively, you can also install the Gateway API CRDs from the upstream project:

kubectl apply -k github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.6.0

Install Flagger in the flagger-system namespace:

kubectl apply -k github.com/fluxcd/flagger//kustomize/gatewayapi

Create a GatewayClass that specifies information about the Gateway controller:

kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
 name: contour
spec:
 controllerName: projectcontour.io/gateway-controller

Create a Gateway that configures load balancing, traffic ACL, etc:

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
 name: contour
 namespace: projectcontour
spec:
 gatewayClassName: contour
 listeners:
 - name: http
 protocol: HTTP
 port: 80
 allowedRoutes:
 namespaces:
 from: All

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services, HTTPRoutes for the Gateway). These objects expose the application inside the mesh and drive the canary analysis and promotion.

Create a test namespace:

kubectl create ns test

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the canary analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create metric templates targeting the Prometheus server in the flagger-system namespace. The PromQL queries below are meant for Envoy, but you can change it to your ingress/mesh provider accordingly.

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: latency
 namespace: flagger-system
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus:9090
 query: |
 histogram_quantile(0.99,
 sum(
 rate(
 envoy_cluster_upstream_rq_time_bucket{
 envoy_cluster_name=~"{{ namespace }}_{{ target }}-canary_[0-9a-zA-Z-]+",
 }[{{ interval }}]
 )
 ) by (le)
 )/1000
---
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: error-rate
 namespace: flagger-system
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus:9090
 query: |
 100 - sum(
 rate(
 envoy_cluster_upstream_rq{
 envoy_cluster_name=~"{{ namespace }}_{{ target }}-canary_[0-9a-zA-Z-]+",
 envoy_response_code!~"5.*"
 }[{{ interval }}]
 )
 )
 /
 sum(
 rate(
 envoy_cluster_upstream_rq{
 envoy_cluster_name=~"{{ namespace }}_{{ target }}-canary_[0-9a-zA-Z-]+",
 }[{{ interval }}]
 )
 )
 * 100

Save the above resource as metric-templates.yaml and then apply it:

kubectl apply -f metric-templates.yaml

Create a canary custom resource (replace “loaclproject.contour.io” with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # service port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 # Gateway API HTTPRoute host names
 hosts:
 - localproject.contour.io
 # Reference to the Gateway that the generated HTTPRoute would attach to.
 gatewayRefs:
 - name: contour
 namespace: projectcontour
 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 10
 metrics:
 - name: error-rate
 # max error rate (5xx responses)
 # percentage (0-100)
 templateRef:
 name: error-rate
 namespace: flagger-system
 thresholdRange:
 max: 1
 interval: 1m
 - name: latency
 templateRef:
 name: latency
 namespace: flagger-system
 # seconds
 thresholdRange:
 max: 0.5
 interval: 30s
 # testing (optional)
 webhooks:
 - name: smoke-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 15s
 metadata:
 type: bash
 cmd: "curl -sd 'anon' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 2m -q 10 -c 2 -host localproject.contour.io http://envoy.projectcontour/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
httproutes.gateway.networking.k8s.io/podinfo

Expose the app outside the cluster

Find the external address of Contour’s Envoy load balancer:

export ADDRESS="$(kubectl -n projectcontour get svc/envoy -ojson \
| jq -r ".status.loadBalancer.ingress[].hostname")"
echo $ADDRESS

Configure your DNS server with a CNAME record (AWS) or A record (GKE/AKS/DOKS) and point a domain e.g. localproject.contour.io to the LB address.

Now you can access the podinfo UI using your domain address.

If you’re using a local cluster via kind/k3s you can port forward the Envoy LoadBalancer service:

kubectl port-forward -n projectcontour svc/envoy 8080:80

Now you can access podinfo via curl -H "Host: localproject.contour.io" localhost:8080

Automated canary promotion

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 0
 Phase: Succeeded
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Scaling up podinfo.test
 Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 2m flagger Advance podinfo.test canary weight 20
 Normal Synced 2m flagger Advance podinfo.test canary weight 25
 Normal Synced 1m flagger Advance podinfo.test canary weight 30
 Normal Synced 1m flagger Advance podinfo.test canary weight 35
 Normal Synced 55s flagger Advance podinfo.test canary weight 40
 Normal Synced 45s flagger Advance podinfo.test canary weight 45
 Normal Synced 35s flagger Advance podinfo.test canary weight 50
 Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
 Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

A canary deployment is triggered by changes in any of the following objects:

Deployment PodSpec (container image, command, ports, env, resources, etc)
ConfigMaps mounted as volumes or mapped to environment variables
Secrets mounted as volumes or mapped to environment variables

You can monitor how Flagger progressively changes the weights of the HTTPRoute object that is attahed to the Gateway with:

watch kubectl get httproute -n test podinfo -o=jsonpath='{.spec.rules}'

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 15 2022-01-16T14:05:07Z
prod frontend Succeeded 0 2022-01-15T16:15:07Z
prod backend Failed 0 2022-01-14T17:05:07Z

Automated rollback

During the canary analysis you can generate HTTP 500 errors and high latency to test if Flagger pauses the rollout.

Trigger another canary deployment:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:6.0.2

Exec into the load tester pod with:

kubectl -n test exec -it flagger-loadtester-xx-xx sh

Generate HTTP 500 errors:

watch curl http://podinfo-canary:9898/status/500

Generate latency:

watch curl http://podinfo-canary:9898/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Canary Weight: 0
 Failed Checks: 10
 Phase: Failed
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger Starting canary deployment for podinfo.test
 Normal Synced 3m flagger Advance podinfo.test canary weight 5
 Normal Synced 3m flagger Advance podinfo.test canary weight 10
 Normal Synced 3m flagger Advance podinfo.test canary weight 15
 Normal Synced 3m flagger Halt podinfo.test advancement error rate 69.17% > 1%
 Normal Synced 2m flagger Halt podinfo.test advancement error rate 61.39% > 1%
 Normal Synced 2m flagger Halt podinfo.test advancement error rate 55.06% > 1%
 Normal Synced 2m flagger Halt podinfo.test advancement error rate 47.00% > 1%
 Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement error rate 38.08% > 1%
 Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
 Warning Synced 1m flagger Canary failed! Scaling down podinfo.test

A/B Testing

Create a canary custom resource (replace “loaclproject.contour.io” with your own domain):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before it is rollback (default 600s)
 progressDeadlineSeconds: 60
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 # service port number
 port: 9898
 # container port number or name (optional)
 targetPort: 9898
 # Gateway API HTTPRoute host names
 hosts:
 - localproject.contour.io
 # Reference to the Gateway that the generated HTTPRoute would attach to.
 gatewayRefs:
 - name: contour
 namespace: projectcontour
 analysis:
 # schedule interval (default 60s)
 interval: 1m
 # max number of failed metric checks before rollback
 threshold: 5
 # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 10
 metrics:
 - name: error-rate
 # max error rate (5xx responses)
 # percentage (0-100)
 templateRef:
 name: error-rate
 namespace: flagger-system
 thresholdRange:
 max: 1
 interval: 1m
 - name: latency
 templateRef:
 name: latency
 namespace: flagger-system
 # seconds
 thresholdRange:
 max: 0.5
 interval: 30s
 # testing (optional)
 webhooks:
 - name: smoke-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 15s
 metadata:
 type: bash
 cmd: "curl -sd 'anon' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 cmd: "hey -z 2m -q 10 -c 2 -host localproject.contour.io -H 'X-Canary: insider' http://envoy.projectcontour/"

The above configuration will run an analysis for ten minutes targeting those users that have an insider cookie.

Save the above resource as podinfo-ab-canary.yaml and then apply it:

kubectl apply -f ./podinfo-ab-canary.yaml

Trigger a canary deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=stefanprodan/podinfo:6.0.3

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/abtest

Status:
 Failed Checks: 0
 Phase: Succeeded
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Scaling up podinfo.test
 Warning Synced 3m flagger Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
 Normal Synced 2m flagger Advance podinfo.test canary iteration 4/10
 Normal Synced 2m flagger Advance podinfo.test canary iteration 5/10
 Normal Synced 1m flagger Advance podinfo.test canary iteration 6/10
 Normal Synced 1m flagger Advance podinfo.test canary iteration 7/10
 Normal Synced 55s flagger Advance podinfo.test canary iteration 8/10
 Normal Synced 45s flagger Advance podinfo.test canary iteration 9/10
 Normal Synced 35s flagger Advance podinfo.test canary iteration 10/10
 Normal Synced 25s flagger Copying podinfo.test template spec to podinfo-primary.test
 Warning Synced 15s flagger Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
 Normal Synced 5s flagger Promotion completed! Scaling down podinfo.test

The above procedures can be extended with custom metrics checks, webhooks, manual promotion approval and Slack or MS Teams notifications.

Flagger: Blue/Green Deployments

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to automate Blue/Green deployments with Flagger and Kubernetes.

For applications that are not deployed on a service mesh, Flagger can orchestrate Blue/Green style deployments with Kubernetes L4 networking. When using a service mesh blue/green can be used as specified here.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer.

Install Flagger and the Prometheus add-on:

helm repo add flagger https://flagger.app

helm upgrade -i flagger flagger/flagger \
--namespace flagger \
--set prometheus.install=true \
--set meshProvider=kubernetes

If you already have a Prometheus instance running in your cluster, you can point Flagger to the ClusterIP service with:

helm upgrade -i flagger flagger/flagger \
--namespace flagger \
--set metricsServer=http://prometheus.monitoring:9090

Optionally you can enable Slack notifications:

helm upgrade -i flagger flagger/flagger \
--reuse-values \
--namespace flagger \
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
--set slack.channel=some-channel-name \
--set slack.user=flagger

Bootstrap

Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployment and ClusterIP services). These objects expose the application inside the cluster and drive the canary analysis and Blue/Green promotion.

Create a test namespace:

kubectl create ns test

Create a deployment and a horizontal pod autoscaler:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Deploy the load testing service to generate traffic during the analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a canary custom resource:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 # service mesh provider can be: kubernetes, istio, appmesh, nginx, gloo
 provider: kubernetes
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # the maximum time in seconds for the canary deployment
 # to make progress before rollback (default 600s)
 progressDeadlineSeconds: 60
 # HPA reference (optional)
 autoscalerRef:
 apiVersion: autoscaling/v2beta2
 kind: HorizontalPodAutoscaler
 name: podinfo
 service:
 port: 9898
 portDiscovery: true
 analysis:
 # schedule interval (default 60s)
 interval: 30s
 # max number of failed checks before rollback
 threshold: 2
 # number of checks to run before rollback
 iterations: 10
 # Prometheus checks based on 
 # http_request_duration_seconds histogram
 metrics:
 - name: request-success-rate
 # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min: 99
 interval: 1m
 - name: request-duration
 # maximum req duration P99
 # milliseconds
 thresholdRange:
 max: 500
 interval: 30s
 # acceptance/load testing hooks
 webhooks:
 - name: smoke-test
 type: pre-rollout
 url: http://flagger-loadtester.test/
 timeout: 15s
 metadata:
 type: bash
 cmd: "curl -sd 'anon' http://podinfo-canary.test:9898/token | grep token"
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test:9898/"

The above configuration will run an analysis for five minutes.

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied 
deployment.apps/podinfo
horizontalpodautoscaler.autoscaling/podinfo
canary.flagger.app/podinfo

# generated 
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary

Blue/Green scenario:

on bootstrap, Flagger will create three ClusterIP services (app-primary,app-canary, app)

and a shadow deployment named app-primary that represents the blue version
when a new version is detected, Flagger would scale up the green version and run the conformance tests

(the tests should target the app-canary ClusterIP service to reach the green version)
if the conformance tests are passing, Flagger would start the load tests and validate them with custom Prometheus queries
if the load test analysis is successful, Flagger will promote the new version to app-primary and scale down the green version

Automated Blue/Green promotion

Trigger a deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Events:

New revision detected podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Pre-rollout check acceptance-test passed
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 100 2019-06-16T14:05:07Z
prod frontend Succeeded 0 2019-06-15T16:15:07Z
prod backend Failed 0 2019-06-14T17:05:07Z

Automated rollback

During the analysis you can generate HTTP 500 errors and high latency to test Flagger’s rollback.

Exec into the load tester pod with:

kubectl -n test exec -it flagger-loadtester-xx-xx sh

Generate HTTP 500 errors:

watch curl http://podinfo-canary.test:9898/status/500

Generate latency:

watch curl http://podinfo-canary.test:9898/delay/1

When the number of failed checks reaches the analysis threshold, the green version is scaled to zero and the rollout is marked as failed.

kubectl -n test describe canary/podinfo

Status:
 Failed Checks: 2
 Phase: Failed
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Synced 3m flagger New revision detected podinfo.test
 Normal Synced 3m flagger Advance podinfo.test canary iteration 1/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 2/10
 Normal Synced 3m flagger Advance podinfo.test canary iteration 3/10
 Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
 Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
 Warning Synced 2m flagger Rolling back podinfo.test failed checks threshold reached 2
 Warning Synced 1m flagger Canary failed! Scaling down podinfo.test

Custom metrics

The analysis can be extended with Prometheus queries. The demo app is instrumented with Prometheus so you can create a custom check that will use the HTTP request duration histogram to validate the canary (green version).

Create a metric template and apply it on the cluster:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: not-found-percentage
 namespace: test
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus.flagger:9090
 query: |
 100 - sum(
 rate(
 http_request_duration_seconds_count{
 kubernetes_namespace="{{ namespace }}",
 kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
 status!="{{ interval }}"
 }[1m]
 )
 )
 /
 sum(
 rate(
 http_request_duration_seconds_count{
 kubernetes_namespace="{{ namespace }}",
 kubernetes_pod_name=~"{{ target }}-[0-9a-zA-Z]+(-[0-9a-zA-Z]+)"
 }[{{ interval }}]
 )
 ) * 100

Edit the canary analysis and add the following metric:

 analysis:
 metrics:
 - name: "404s percentage"
 templateRef:
 name: not-found-percentage
 thresholdRange:
 max: 5
 interval: 1m

The above configuration validates the canary (green version) by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the rollout is rolled back.

Trigger a deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.3

Generate 404s:

watch curl http://podinfo-canary.test:9898/status/400

Watch Flagger logs:

kubectl -n flagger logs deployment/flagger -f | jq .msg

New revision detected podinfo.test
Scaling up podinfo.test
Advance podinfo.test canary iteration 1/10
Halt podinfo.test advancement 404s percentage 6.20 > 5
Halt podinfo.test advancement 404s percentage 6.45 > 5
Rolling back podinfo.test failed checks threshold reached 2
Canary failed! Scaling down podinfo.test

If you have alerting configured, Flagger will send a notification with the reason why the canary failed.

Conformance Testing with Helm

Flagger comes with a testing service that can run Helm tests when configured as a pre-rollout webhook.

Deploy the Helm test runner in the kube-system namespace using the tiller service account:

helm repo add flagger https://flagger.app

helm upgrade -i flagger-helmtester flagger/loadtester \
--namespace=kube-system \
--set serviceAccountName=tiller

When deployed the Helm tester API will be available at http://flagger-helmtester.kube-system/.

Add a helm test pre-rollout hook to your chart:

 analysis:
 webhooks:
 - name: "conformance testing"
 type: pre-rollout
 url: http://flagger-helmtester.kube-system/
 timeout: 3m
 metadata:
 type: "helm"
 cmd: "test {{ .Release.Name }} --cleanup"

When the canary analysis starts, Flagger will call the pre-rollout webhooks. If the helm test fails, Flagger will retry until the analysis threshold is reached and the canary is rolled back.

For an in-depth look at the analysis process read the usage docs.

Flagger: Canary analysis with Prometheus Operator

Mon, 01 Jan 0001 00:00:00 +0000

This guide show you how to use Prometheus Operator for canary analysis.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer and Prometheus Operator v0.40 or newer.

Install Prometheus Operator with Helm v3:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

kubectl create ns monitoring
helm upgrade -i prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set fullnameOverride=prometheus

The prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false option allows Prometheus Operator to watch serviceMonitors outside of its namespace.

Install Flagger by setting the metrics server to Prometheus:

helm repo add flagger https://flagger.app

kubectl create ns flagger-system
helm upgrade -i flagger flagger/flagger \
--namespace flagger-system \
--set metricsServer=http://prometheus-prometheus.monitoring:9090 \
--set meshProvider=kubernetes

Install Flagger’s tester:

helm upgrade -i loadtester flagger/loadtester \
--namespace flagger-system

Install podinfo demo app:

helm repo add podinfo https://stefanprodan.github.io/podinfo

kubectl create ns test
helm upgrade -i podinfo podinfo/podinfo \
--namespace test \
--set service.enabled=false

Service monitors

The demo app is instrumented with Prometheus, so you can create a ServiceMonitor objects to scrape podinfo’s metrics endpoint:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: podinfo-canary
 namespace: test
spec:
 endpoints:
 - path: /metrics
 port: http
 interval: 5s
 selector:
 matchLabels:
 app: podinfo-canary
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: podinfo-primary
 namespace: test
spec:
 endpoints:
 - path: /metrics
 port: http
 interval: 5s
 selector:
 matchLabels:
 app: podinfo

We are setting interval: 5s to have a more aggressive scraping. If you do not define it, you should use a longer interval in the Canary object.

Metric templates

Create a metric template to measure the HTTP requests error rate:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: error-rate
 namespace: test
spec:
 provider:
 address: http://prometheus-prometheus.monitoring:9090
 type: prometheus
 query: |
 100 - rate(
 http_requests_total{
 namespace="{{ namespace }}",
 job="{{ target }}-canary",
 status!~"5.*"
 }[{{ interval }}])
 /
 rate(
 http_requests_total{
 namespace="{{ namespace }}",
 job="{{ target }}-canary"
 }[{{ interval }}]
 ) * 100

Create a metric template to measure the HTTP requests average duration:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: latency
 namespace: test
spec:
 provider:
 address: http://prometheus-prometheus.monitoring:9090
 type: prometheus
 query: |
 histogram_quantile(0.99,
 sum(
 rate(
 http_request_duration_seconds_bucket{
 namespace="{{ namespace }}",
 job="{{ target }}-canary"
 }[{{ interval }}]
 )
 ) by (le)
 )

Canary analysis

Using the metrics template you can configure the canary analysis with HTTP error rate and latency checks:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: kubernetes
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 progressDeadlineSeconds: 60
 service:
 port: 80
 targetPort: http
 name: podinfo
 analysis:
 interval: 30s
 iterations: 10
 threshold: 2
 metrics:
 - name: error-rate
 templateRef:
 name: error-rate
 thresholdRange:
 max: 1
 interval: 30s
 - name: latency
 templateRef:
 name: latency
 thresholdRange:
 max: 0.5
 interval: 30s
 webhooks:
 - name: load-test
 type: rollout
 url: "http://loadtester.flagger-system/"
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test/"

Based on the above specification, Flagger creates the primary and canary Kubernetes ClusterIP service.

During the canary analysis, Prometheus will scrape the canary service and Flagger will use the HTTP error rate and latency queries to determine if the release should be promoted or rolled back.

Flagger: Canary analysis with KEDA ScaledObjects

Mon, 01 Jan 0001 00:00:00 +0000

This guide shows you how to use Flagger with KEDA ScaledObjects to autoscale workloads during a Canary analysis run. We will be using a Blue/Green deployment strategy with the Kubernetes provider for the sake of this tutorial, but you can use any deployment strategy combined with any supported provider.

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or newer. For this tutorial, we’ll need KEDA 2.7.1 or newer.

Install KEDA:

helm repo add kedacore https://kedacore.github.io/charts
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda

Install Flagger:

helm repo add flagger https://flagger.app

helm upgrade -i flagger flagger/flagger \
--namespace flagger \
--set prometheus.install=true \
--set meshProvider=kubernetes

Bootstrap

Flagger takes a Kubernetes deployment and a KEDA ScaledObject targeting the deployment. It then creates a series of objects (Kubernetes deployments, ClusterIP services and another KEDA ScaledObject targeting the created Deployment). These objects expose the application inside the mesh and drive the Canary analysis and Blue/Green promotion.

Create a test namespace:

kubectl create ns test

Create a deployment named podinfo:

kubectl apply -n test -f https://raw.githubusercontent.com/fluxcd/flagger/main/kustomize/podinfo/deployment.yaml

Deploy the load testing service to generate traffic during the analysis:

kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a ScaledObject which targets the podinfo deployment and uses Prometheus as a trigger:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: podinfo-so
 namespace: test
spec:
 scaleTargetRef:
 name: podinfo
 pollingInterval: 10
 cooldownPeriod: 20
 minReplicaCount: 1
 maxReplicaCount: 3
 triggers:
 - type: prometheus
 metadata:
 name: prom-trigger
 serverAddress: http://flagger-prometheus.flagger-system:9090
 metricName: http_requests_total
 query: sum(rate(http_requests_total{ app="podinfo" }[30s]))
 threshold: '5'

Create a canary custom resource for the podinfo deployment:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
 name: podinfo
 namespace: test
spec:
 provider: kubernetes
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 # Scaler reference
 autoscalerRef:
 apiVersion: keda.sh/v1alpha1
 kind: ScaledObject
 # ScaledObject targeting the canary deployment
 name: podinfo-so
 # Mapping between trigger names and the related query to use for the generated 
 # ScaledObject targeting the primary deployment. (Optional)
 primaryScalerQueries:
 prom-trigger: sum(rate(http_requests_total{ app="podinfo-primary" }[30s]))
 # Overriding replica scaling configuration for the generated ScaledObject
 # targeting the primary deployment. (Optional)
 primaryScalerReplicas:
 minReplicas: 2
 maxReplicas: 5
 # the maximum time in seconds for the canary deployment
 # to make progress before rollback (default 600s)
 progressDeadlineSeconds: 60
 service:
 port: 80
 targetPort: 9898
 name: podinfo-svc
 portDiscovery: true
 analysis:
 # schedule interval (default 60s)
 interval: 15s
 # max number of failed checks before rollback
 threshold: 5
 # number of checks to run before promotion
 iterations: 5
 # Prometheus checks based on 
 # http_request_duration_seconds histogram
 metrics:
 - name: request-success-rate
 interval: 1m
 thresholdRange:
 min: 99
 - name: request-duration
 interval: 30s
 thresholdRange:
 max: 500
 # load testing hooks
 webhooks:
 - name: load-test
 url: http://flagger-loadtester.test/
 timeout: 5s
 metadata:
 type: cmd
 cmd: "hey -z 2m -q 20 -c 2 http://podinfo-svc-canary.test/"

Save the above resource as podinfo-canary.yaml and then apply it:

kubectl apply -f ./podinfo-canary.yaml

After a couple of seconds Flagger will create the canary objects:

# applied
deployment.apps/podinfo
scaledobject.keda.sh/podinfo-so
canary.flagger.app/podinfo

# generated
deployment.apps/podinfo-primary
horizontalpodautoscaler.autoscaling/podinfo-primary
service/podinfo
service/podinfo-canary
service/podinfo-primary
scaledobject.keda.sh/podinfo-so-primary

We refer to our ScaledObject for the canary deployment using .spec.autoscalerRef. Flagger will use this to generate a ScaledObject which will scale the primary deployment. By default, Flagger will try to guess the query to use for the primary ScaledObject, by replacing all mentions of .spec.targetRef.Name and {.spec.targetRef.Name}-canary with {.spec.targetRef.Name}-primary, for all triggers. For eg, if your ScaledObject has a trigger query defined as: sum(rate(http_requests_total{ app="podinfo" }[30s])) or sum(rate(http_requests_total{ app="podinfo-primary" }[30s])), then the primary ScaledObject will have the same trigger with a query defined as sum(rate(http_requests_total{ app="podinfo-primary" }[30s])).

If, the generated query does not meet your requirements, you can specify the query for autoscaling the primary deployment explicitly using .spec.autoscalerRef.primaryScalerQueries, which lets you define a query for each trigger. Please note that, your ScaledObject’s .spec.triggers[@].name must not be blank, as Flagger needs that to identify each trigger uniquely.

In the situation when it is desired to have different scaling replica configuration between the canary and primary deployment ScaledObject you can use the .spec.autoscalerRef.primaryScalerReplicas to override these values for the generated primary ScaledObject.

After the bootstrap, the podinfo deployment will be scaled to zero and the traffic to podinfo.test will be routed to the primary pods. To keep the podinfo deployment at 0 replicas and pause auto scaling, Flagger will add an annotation to your ScaledObject: autoscaling.keda.sh/paused-replicas: 0. During the canary analysis, the annotation is removed, to enable auto scaling for the podinfo deployment. The podinfo-canary.test address can be used to target directly the canary pods. When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary. The Blue/Green deployment will run for five iterations while validating the HTTP metrics and rollout hooks every 15 seconds.

Automated Blue/Green promotion

Trigger a deployment by updating the container image:

kubectl -n test set image deployment/podinfo \
podinfod=ghcr.io/stefanprodan/podinfo:6.0.1

Flagger detects that the deployment revision changed and starts a new rollout:

kubectl -n test describe canary/podinfo

Events:

New revision detected podinfo.test
Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
Pre-rollout check acceptance-test passed
Advance podinfo.test canary iteration 1/10
Advance podinfo.test canary iteration 2/10
Advance podinfo.test canary iteration 3/10
Advance podinfo.test canary iteration 4/10
Advance podinfo.test canary iteration 5/10
Advance podinfo.test canary iteration 6/10
Advance podinfo.test canary iteration 7/10
Advance podinfo.test canary iteration 8/10
Advance podinfo.test canary iteration 9/10
Advance podinfo.test canary iteration 10/10
Copying podinfo.test template spec to podinfo-primary.test
Waiting for podinfo-primary.test rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to the deployment during the canary analysis, Flagger will restart the analysis.

You can monitor all canaries with:

watch kubectl get canaries --all-namespaces

NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
test podinfo Progressing 100 2019-06-16T14:05:07Z

You can monitor the scaling of the deployments with:

watch kubectl -n test get deploy podinfo
NAME READY UP-TO-DATE AVAILABLE AGE
flagger-loadtester 1/1 1 1 4m21s
podinfo 3/3 3 3 4m28s
podinfo-primary 3/3 3 3 3m14s

You can mointor how Flagger edits the annotations of your ScaledObject with:

watch "kubectl get -n test scaledobjects podinfo-so -o=jsonpath='{.metadata.annotations}'"

Flagger: Zero downtime deployments

Mon, 01 Jan 0001 00:00:00 +0000

This is a list of things you should consider when dealing with a high traffic production environment if you want to minimise the impact of rolling updates and downscaling.

Deployment strategy

Limit the number of unavailable pods during a rolling update:

apiVersion: apps/v1
kind: Deployment
spec:
 progressDeadlineSeconds: 120
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxUnavailable: 0

The default progress deadline for a deployment is ten minutes. You should consider adjusting this value to make the deployment process fail faster.

Liveness health check

You application should expose a HTTP endpoint that Kubernetes can call to determine if your app transitioned to a broken state from which it can’t recover and needs to be restarted.

livenessProbe:
 exec:
 command:
 - wget
 - --quiet
 - --tries=1
 - --timeout=4
 - --spider
 - http://localhost:8080/healthz
 timeoutSeconds: 5
 initialDelaySeconds: 5

If you’ve enabled mTLS, you’ll have to use exec for liveness and readiness checks since kubelet is not part of the service mesh and doesn’t have access to the TLS cert.

Readiness health check

You application should expose a HTTP endpoint that Kubernetes can call to determine if your app is ready to receive traffic.

readinessProbe:
 exec:
 command:
 - wget
 - --quiet
 - --tries=1
 - --timeout=4
 - --spider
 - http://localhost:8080/readyz
 timeoutSeconds: 5
 initialDelaySeconds: 5
 periodSeconds: 5

If your app depends on external services, you should check if those services are available before allowing Kubernetes to route traffic to an app instance. Keep in mind that the Envoy sidecar can have a slower startup than your app. This means that on application start you should retry for at least a couple of seconds any external connection.

Graceful shutdown

Before a pod gets terminated, Kubernetes sends a SIGTERM signal to every container and waits for period of time (30s by default) for all containers to exit gracefully. If your app doesn’t handle the SIGTERM signal or if it doesn’t exit within the grace period, Kubernetes will kill the container and any inflight requests that your app is processing will fail.

apiVersion: apps/v1
kind: Deployment
spec:
 template:
 spec:
 terminationGracePeriodSeconds: 60
 containers:
 - name: app
 lifecycle:
 preStop:
 exec:
 command:
 - sleep
 - "10"

Your app container should have a preStop hook that delays the container shutdown. This will allow the service mesh to drain the traffic and remove this pod from all other Envoy sidecars before your app becomes unavailable.

Delay Envoy shutdown

Even if your app reacts to SIGTERM and tries to complete the inflight requests before shutdown, that doesn’t mean that the response will make it back to the caller. If the Envoy sidecar shuts down before your app, then the caller will receive a 503 error.

To mitigate this issue you can add a preStop hook to the Istio proxy and wait for the main app to exit before Envoy exits.

#!/bin/bash
set -e
if ! pidof envoy &>/dev/null; then
 exit 0
fi

if ! pidof pilot-agent &>/dev/null; then
 exit 0
fi

while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do
 sleep 1;
done

exit 0

You’ll have to build your own Envoy docker image with the above script and modify the Istio injection webhook with the preStop directive.

Thanks to Stono for his excellent tips on minimising 503s.

Resource requests and limits

Setting CPU and memory requests/limits for all workloads is a mandatory step if you’re running a production system. Without limits your nodes could run out of memory or become unresponsive due to CPU exhausting. Without CPU and memory requests, the Kubernetes scheduler will not be able to make decisions about which nodes to place pods on.

apiVersion: apps/v1
kind: Deployment
spec:
 template:
 spec:
 containers:
 - name: app
 resources:
 limits:
 cpu: 1000m
 memory: 1Gi
 requests:
 cpu: 100m
 memory: 128Mi

Note that without resource requests the horizontal pod autoscaler can’t determine when to scale your app.

Autoscaling

A production environment should be able to handle traffic bursts without impacting the quality of service. This can be achieved with Kubernetes autoscaling capabilities. Autoscaling in Kubernetes has two dimensions: the Cluster Autoscaler that deals with node scaling operations and the Horizontal Pod Autoscaler that automatically scales the number of pods in a deployment.

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: app
 minReplicas: 2
 maxReplicas: 4
 metrics:
 - type: Resource
 resource:
 name: cpu
 targetAverageValue: 900m
 - type: Resource
 resource:
 name: memory
 targetAverageValue: 768Mi

The above HPA ensures your app will be scaled up before the pods reach the CPU or memory limits.

Ingress retries

To minimise the impact of downscaling operations you can make use of Envoy retry capabilities.

apiVersion: flagger.app/v1beta1
kind: Canary
spec:
 service:
 port: 9898
 gateways:
 - public-gateway.istio-system.svc.cluster.local
 hosts:
 - app.example.com
 retries:
 attempts: 10
 perTryTimeout: 5s
 retryOn: "gateway-error,connect-failure,refused-stream"

When the HPA scales down your app, your users could run into 503 errors. The above configuration will make Envoy retry the HTTP requests that failed due to gateway errors.