Flux – Notification Controller

Flux: Controller Options

Mon, 01 Jan 0001 00:00:00 +0000

To customise the controller options at install time, please see the bootstrap customization guide.

Flags

Name	Type	Description
`--concurrent`	int	The number of concurrent notification reconciles. (default 4)
`--default-service-account`	string	Default service account to use for workload identity when not specified in resources.
`--enable-leader-election`	boolean	Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
`--events-addr`	string	The address of the events receiver.
`--health-addr`	string	The address the health endpoint binds to. (default “:9440”)
`--leader-election-lease-duration`	duration	Interval at which non-leader candidates will wait to force acquire leadership (duration string). (default 35s)
`--leader-election-release-on-cancel`	boolean	Defines if the leader should step down voluntarily on controller manager shutdown. (default true)
`--leader-election-renew-deadline`	duration	Duration that the leading controller manager will retry refreshing leadership before giving up (duration string). (default 30s)
`--leader-election-retry-period`	duration	Duration the LeaderElector clients should wait between tries of actions (duration string). (default 5s)
`--log-encoding`	string	Log encoding format. Can be ‘json’ or ‘console’. (default “json”)
`--log-level`	string	Log verbosity level. Can be one of ’trace’, ‘debug’, ‘info’, ’error’. (default “info”)
`--metrics-addr`	string	The address the metric endpoint binds to. (default “:8080”)
`--no-cross-namespace-refs`	boolean	When set to true, references between custom resources are allowed only if the reference and the referee are in the same namespace.
`--rate-limit-interval`	duration	Interval in which rate limit has effect. (default 5m0s)
`--receiverAddr`	string	The address the webhook receiver endpoint binds to. (default “:9292”)
`--token-cache-max-size`	int	The maximum amount of entries in the LRU cache used for tokens. (default 100, enabled)
`--token-cache-max-duration`	duration	The maximum duration for which a token would be considered unexpired. This is capped at 1h. (default 1h)
`--watch-all-namespaces`	boolean	Watch for custom resources in all namespaces, if set to false it will only watch the runtime namespace. (default true)
`--feature-gates`	mapStringBool	A comma separated list of key=value pairs defining the state of experimental features.

Feature Gates

Name	Default Value	Description
`CacheSecretsAndConfigMaps`	`false`	Configures the caching of Secrets and ConfigMaps by the controller-runtime client. When enabled, it will cache both object types, resulting in increased memory usage and cluster-wide RBAC permissions (list and watch).
`DisableConfigWatchers`	`false`	Disables the watchers for ConfigMaps and Secrets.
`ObjectLevelWorkloadIdentity`	`false`	Enables the use of object-level workload identity for the controller.

Flux: Alerts

Mon, 01 Jan 0001 00:00:00 +0000

The Alert API defines how events are filtered by severity and involved object, and what provider to use for dispatching.

Example

The following is an example of how to send alerts to Slack when Flux fails to reconcile the flux-system namespace.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: slack-bot
 namespace: flux-system
spec:
 type: slack
 channel: general
 address: https://slack.com/api/chat.postMessage
 secretRef:
 name: slack-bot-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: slack
 namespace: flux-system
spec:
 providerRef:
 name: slack-bot
 eventMetadata:
 summary: Cluster addons impacted
 env: prod
 cluster: my-cluster
 region: us-east-2
 eventSeverity: error
 eventSources:
 - kind: GitRepository
 name: '*'
 - kind: Kustomization
 name: '*'

In the above example:

A Provider named slack-bot is created, indicated by the Provider.metadata.name field.
An Alert named slack is created, indicated by the Alert.metadata.name field.
The Alert references the slack-bot provider, indicated by the Alert.spec.providerRef field.
The notification-controller starts listening for events sent for all GitRepositories and Kustomizations in the flux-system namespace.
When an event with severity error is received, the controller posts a message on Slack channel from .spec.channel, containing the metadata and the reconciliation error.

You can run this example by saving the manifests into slack-alerts.yaml.

First create a secret with the Slack bot token:

kubectl -n flux-system create secret generic slack-bot-token --from-literal=token=xoxb-YOUR-TOKEN

Apply the resources on the cluster:

kubectl -n flux-system apply --server-side -f slack-alerts.yaml

Writing an Alert spec

As with all other Kubernetes config, an Alert needs apiVersion, kind, and metadata fields. The name of an Alert object must be a valid DNS subdomain name.

An Alert also needs a .spec section.

Summary

.spec.summary is an optional field to specify a short description of the impact.

The summary max length can’t be greater than 255 characters.

Warning: Support for .spec.summary has been deprecated and will be removed in Alert API v1 GA. If you have any Alerts using this field, the controller will log a deprecation warning. Please use .spec.eventMetadata.summary or object annotations for defining alert summary instead.

Provider reference

.spec.providerRef.name is a required field to specify a name reference to a Provider in the same namespace as the Alert.

Event sources

.spec.eventSources is a required field to specify a list of references to Flux objects for which events are forwarded to the alert provider API.

To select events issued by Flux objects, each entry in the .spec.eventSources list must contain the following fields:

kind is the Flux Custom Resource Kind such as GitRepository, HelmRelease, Kustomization, etc.
name is the Flux Custom Resource .metadata.name, or it can be set to the * wildcard.
namespace is the Flux Custom Resource .metadata.namespace. When not specified, the Alert .metadata.namespace is used instead.

Select objects by name

To select events issued by a single Flux object, set the kind, name and namespace:

eventSources:
 - kind: GitRepository
 name: webapp
 namespace: apps

Select all objects in a namespace

The * wildcard can be used to select events issued by all Flux objects of a particular kind in a namespace:

eventSources:
 - kind: HelmRelease
 name: '*'
 namespace: apps

Select objects by label

To select events issued by all Flux objects of a particular kind with specific labels:

eventSources:
 - kind: HelmRelease
 name: '*'
 namespace: apps
 matchLabels:
 team: app-dev

Disable cross-namespace selectors

Note: On multi-tenant clusters, platform admins can disable cross-namespace references by starting the controller with the --no-cross-namespace-refs=true flag. When this flag is set, alerts can only refer to event sources in the same namespace as the alert object, preventing tenants from subscribing to another tenant’s events.

Event metadata

.spec.eventMetadata is an optional field for adding metadata to events dispatched by the controller. This can be used for enhancing the context of the event, e.g. with cluster-level information.

For all the event metadata sources and their precedence order, please refer to Event metadata from object annotations.

Example

Add metadata fields to successful HelmRelease events:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: <name>
spec:
 eventSources:
 - kind: HelmRelease
 name: '*'
 inclusionList:
 - ".*succeeded.*"
 eventMetadata:
 env: production
 cluster: my-cluster
 region: us-east-1

Event metadata from object annotations

Event metadata has four sources. They are listed below in order of precedence, from lowest to highest:

User-defined metadata on Flux objects, set with the event.toolkit.fluxcd.io/ prefix in the keys of the object’s .metadata.annotations.
User-defined metadata on the Alert object, set with .spec.eventMetadata.
User-defined summary on the Alert object, set with .spec.summary (deprecated, see docs).
Controller-defined metadata, set with the <controller group>.toolkit.fluxcd.io/ prefix in the metadata keys of the event payload.

If there are any metadata key conflicts between the sources, the higher precedence source will override the lower precedence source, and a warning log and Kubernetes event will be emitted.

Example

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: <name>
spec:
 eventSources:
 - kind: HelmRelease
 name: '*'
 eventMetadata:
 env: production
 cluster: my-cluster
 region: us-east-1
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-webapp
 annotations:
 event.toolkit.fluxcd.io/summary: "my-webapp impacted. Playbook: <URL to playbook>"
 event.toolkit.fluxcd.io/deploymentID: e076e315-5a48-41c3-81c8-8d8bdee7d74d
spec:
 ... # fields omitted for brevity

In the above example, the event payload dispatched by the controller will look like this (most fields omitted for highlighting the metadata):

{
 "metadata": {
 "env": "production",
 "cluster": "my-cluster",
 "region": "us-east-1",
 "summary": "my-webapp impacted. Playbook: <URL to playbook>",
 "deploymentID": "e076e315-5a48-41c3-81c8-8d8bdee7d74d"
 }
}

Event severity

.spec.eventSeverity is an optional field to filter events based on severity. When not specified, or when the value is set to info, all events are forwarded to the alert provider API, including errors. To receive alerts only on errors, set the field value to error.

Event exclusion

.spec.exclusionList is an optional field to specify a list of regex expressions to filter events based on message content. The event will be excluded if the message matches at least one of the expressions in the list.

Example

Skip alerting if the message matches a Go regex from the exclusion list:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: <name>
spec:
 eventSources:
 - kind: GitRepository
 name: '*'
 exclusionList:
 - "waiting.*socket"

The above definition will not send alerts for transient Git clone errors like:

unable to clone 'ssh://git@ssh.dev.azure.com/v3/...', error: SSH could not read data: Error waiting on socket

Event inclusion

.spec.inclusionList is an optional field to specify a list of regex expressions to filter events based on message content. The event will be sent if the message matches at least one of the expressions in the list, and discarded otherwise. If the message matches one of the expressions in the inclusion list but also matches one of the expressions in the exclusion list, then the event is still discarded (exclusion is stronger than inclusion).

Example

Alert if the message matches a Go regex from the inclusion list:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: <name>
spec:
 eventSources:
 - kind: HelmRelease
 name: '*'
 inclusionList:
 - ".*succeeded.*"
 exclusionList:
 - ".*uninstall.*"
 - ".*test.*"

The above definition will send alerts for successful Helm installs, upgrades and rollbacks, but not uninstalls and tests.

Suspend

.spec.suspend is an optional field to suspend the altering. When set to true, the controller will stop processing events. When the field is set to false or removed, it will resume.

Flux: Events

Mon, 01 Jan 0001 00:00:00 +0000

The Event API defines the structure of the events issued by Flux controllers.

Flux controllers use the fluxcd/pkg/runtime/events package to push events to the notification-controller API.

Example

The following is an example of an event sent by kustomize-controller to report a reconciliation error.

{
 "involvedObject": {
 "apiVersion": "kustomize.toolkit.fluxcd.io/v1beta2",
 "kind": "Kustomization",
 "name": "webapp",
 "namespace": "apps",
 "uid": "7d0cdc51-ddcf-4743-b223-83ca5c699632"
 },
 "metadata": {
 "kustomize.toolkit.fluxcd.io/revision": "main/731f7eaddfb6af01cb2173e18f0f75b0ba780ef1"
 },
 "severity":"error",
 "reason": "ValidationFailed",
 "message":"service/apps/webapp validation error: spec.type: Unsupported value: Ingress",
 "reportingController":"kustomize-controller",
 "timestamp":"2022-10-28T07:26:19Z"
}

In the above example:

An event is issued by kustomize-controller for a specific object, indicated in the involvedObject field.
The notification-controller receives the event and finds the alerts that match the involvedObject and severity values.
For all matching alerts, the controller posts the message and the source revision extracted from metadata to the alert provider API.

Event structure

The Go type that defines the event structure can be found in the fluxcd/pkg/apis/event/v1beta1 package.

Rate limiting

Events received by notification-controller are subject to rate limiting to reduce the amount of duplicate alerts sent to external systems like Slack, Sentry, etc.

Events are rate limited based on involvedObject.name, involvedObject.namespace, involvedObject.kind, message, and metadata. The interval of the rate limit is set by default to 5m but can be configured with the --rate-limit-interval controller flag.

The event server exposes HTTP request metrics to track the amount of rate limited events. The following promql will get the rate at which requests are rate limited:

rate(gotk_event_http_request_duration_seconds_count{code="429"}[30s])

Flux: Receivers

Mon, 01 Jan 0001 00:00:00 +0000

The Receiver API defines an incoming webhook receiver that triggers the reconciliation for a group of Flux Custom Resources.

Example

The following is an example of how to configure an incoming webhook for the GitHub repository where Flux was bootstrapped with flux bootstrap github. After a Git push, GitHub will send a push event to notification-controller, which in turn tells Flux to pull and apply the latest changes from upstream.

Note: The following assumes an Ingress exposes the controller’s webhook-receiver Kubernetes Service. How to configure the Ingress is out of scope for this example.

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: github-receiver
 namespace: flux-system
spec:
 type: github
 events:
 - "ping"
 - "push"
 secretRef:
 name: receiver-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: flux-system

In the above example:

A Receiver named github-receiver is created, indicated by the .metadata.name field.
The notification-controller generates a unique webhook path using the Receiver name, namespace and the token from the referenced .spec.secretRef.name secret.
The incoming webhook path is reported in the .status.webhookPath field.
When a GitHub push event is received, the controller verifies the payload’s integrity and authenticity, using HMAC and the X-Hub-Signature HTTP header.
If the event type matches .spec.events and the payload is verified, then the controller triggers a reconciliation for the flux-system GitRepository which is listed under .spec.resources.

You can run this example by saving the manifest into github-receiver.yaml.

Generate a random string and create a Secret with a token field:

TOKEN=$(head -c 12 /dev/urandom | shasum | cut -d ' ' -f1)

kubectl -n flux-system create secret generic receiver-token \
 --from-literal=token=$TOKEN

Apply the resource on the cluster:

kubectl -n flux-system apply -f github-receiver.yaml

Run kubectl -n flux-system describe receiver github-receiver to see its status:

...
Status:
 Conditions:
 Last Transition Time: 2022-11-16T23:43:38Z
 Message: Receiver initialised for path: /hook/bed6d00b5555b1603e1f59b94d7fdbca58089cb5663633fb83f2815dc626d92b
 Observed Generation: 1
 Reason: Succeeded
 Status: True
 Type: Ready
 Observed Generation: 1
 Webhook Path: /hook/bed6d00b5555b1603e1f59b94d7fdbca58089cb5663633fb83f2815dc626d92b
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Succeeded 82s notification-controller Reconciliation finished, next run in 10m

Run kubectl -n flux-system get receivers to see the generated webhook path:

NAME READY STATUS
github-receiver True Receiver initialised for path: /hook/bed6d00b5555b1603e1f59b94d7fdbca58089cb5663633fb83f2815dc626d92b

On GitHub, navigate to your repository and click on the “Add webhook” button under “Settings/Webhooks”. Fill the form with:
- Payload URL: The composed address, consisting of the Ingress’ hostname exposing the controller’s webhook-receiver Kubernetes Service, and the generated path for the Receiver. For this example: https://<hostname>/hook/bed6d00b5555b1603e1f59b94d7fdbca58089cb5663633fb83f2815dc626d92b
- Secret: The token string generated in step 1.

Writing a Receiver spec

As with all other Kubernetes config, a Receiver needs apiVersion, kind, and metadata fields. The name of a Receiver object must be a valid DNS subdomain name.

A Receiver also needs a .spec section.

Type

.spec.type is a required field that specifies how the controller should handle the incoming webhook request.

Supported Receiver types

Receiver	Type	Supports filtering using Events
Generic webhook	`generic`	❌
Generic webhook with HMAC	`generic-hmac`	❌
GitHub	`github`	✅
Gitea	`github`	✅
GitLab	`gitlab`	✅
Bitbucket server	`bitbucket`	✅
Harbor	`harbor`	❌
DockerHub	`dockerhub`	❌
Quay	`quay`	❌
Nexus	`nexus`	❌
Azure Container Registry	`acr`	❌
Google Container Registry	`gcr`	❌
CDEvents	`cdevents`	✅

Generic

When a Receiver’s .spec.type is set to generic, the controller will respond to any HTTP request to the generated .status.webhookPath path, and request a reconciliation for all listed Resources.

Note: This type of Receiver does not perform any validation on the incoming request, and it does not support filtering using Events.

Generic example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: generic-receiver
 namespace: default
spec:
 type: generic
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: webapp
 namespace: default

Generic HMAC

When a Receiver’s .spec.type is set to generic-hmac, the controller will respond to any HTTP request to the generated .status.webhookPath path, while verifying the request’s payload integrity and authenticity using HMAC.

The controller uses the X-Signature header to get the hash signature. This signature should be prefixed with the hash function (sha1, sha256 or sha512) used to generate the signature, in the following format: <hash-function>=<hash>.

To validate the HMAC signature, the controller will use the token string from the Secret reference to generate a hash signature using the same hash function as the one specified in the X-Signature header.

If the generated hash signature matches the one specified in the X-Signature header, the controller will request a reconciliation for all listed Resources.

Note: This type of Receiver does not support filtering using Events.

Generic HMAC example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: generic-hmac-receiver
 namespace: default
spec:
 type: generic-hmac
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: webapp
 namespace: default

HMAC signature generation example

Generate the HMAC hash for the request body using OpenSSL:
```
printf '<request-body>' | openssl dgst -sha1 -r -hmac "<token>" | awk '{print $1}'
```
You can replace the -sha1 flag with -sha256 or -sha512 to use a different hash function.

Send an HTTP POST request with the body and the HMAC hash to the webhook URL:

curl <webhook-url> -X POST -H "X-Signature: <hash-function>=<generated-hash>" -d '<request-body>'

GitHub

When a Receiver’s .spec.type is set to github, the controller will respond to an HTTP webhook event payload from GitHub to the generated .status.webhookPath path, while verifying the payload using HMAC.

The controller uses the X-Hub-Signature header from the request made by GitHub to get the hash signature. To enable the inclusion of this header, the token string from the Secret reference must be configured as the secret token for the webhook.

This type of Receiver offers the ability to filter incoming events by comparing the X-GitHub-Event header to the list of Events. For a list of available events, see the GitHub documentation.

GitHub example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: github-receiver
 namespace: default
spec:
 type: github
 events:
 - "ping"
 - "push"
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: webapp

The above example makes use of the .spec.events field to filter incoming events from GitHub, instructing the controller to only respond to ping and push events.

Gitea

For Gitea, the .spec.type field can be set to github as it produces GitHub type compatible webhook event payloads.

Note: While the payloads are compatible with the GitHub type, the number of available events may be limited and/or different from the ones available in GitHub. Refer to the Gitea source code to see the list of available events.

GitLab

When a Receiver’s .spec.type is set to gitlab, the controller will respond to an HTTP webhook event payload from GitLab to the generated .status.webhookPath path.

The controller validates the payload’s authenticity by comparing the X-Gitlab-Token header from the request made by GitLab to the token string from the Secret reference. To enable the inclusion of this header, the token string must be configured as the “Secret token” while configuring a webhook in GitLab.

If the two tokens match, the controller will request a reconciliation for all listed Resources.

This type of Receiver offers the ability to filter incoming events by comparing the X-Gitlab-Event header to the list of Events. For a list of available webhook types, refer to the GitLab documentation.

GitLab example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: gitlab-receiver
 namespace: default
spec:
 type: gitlab
 events:
 - "Push Hook"
 - "Tag Push Hook"
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: webapp-frontend
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: webapp-backend

The above example makes use of the .spec.events field to filter incoming events from GitLab, instructing the controller to only respond to Push Hook and Tag Push Hook events.

Bitbucket Server

When a Receiver’s .spec.type is set to bitbucket, the controller will respond to an HTTP webhook event payload from Bitbucket Server to the generated .status.webhookPath path, while verifying the payload’s integrity and authenticity using HMAC.

The controller uses the X-Hub-Signature header from the request made by BitBucket Server to get the hash signature. To enable the inclusion of this header, the token string from the Secret reference must be configured as the “Secret” while creating a webhook in Bitbucket Server.

This type of Receiver offers the ability to filter incoming events by comparing the X-Event-Key header to the list of Events. For a list of available event keys, refer to the Bitbucket Server documentation.

Note: Bitbucket Cloud does not support signing webhook requests ( BCLOUD-14683, BCLOUD-12195). If your repositories are on Bitbucket Cloud, you will need to use a Generic Receiver instead.

Bitbucket Server example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: bitbucket-receiver
 namespace: default
spec:
 type: bitbucket
 events:
 - "repo:refs_changed"
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: webapp

The above example makes use of the .spec.events field to filter incoming events from Bitbucket Server, instructing the controller to only respond to repo:refs_changed (Push) events.

Harbor

When a Receiver’s .spec.type is set to harbor, the controller will respond to an HTTP webhook event payload from Harbor to the generated .status.webhookPath path.

The controller validates the payload’s authenticity by comparing the Authorization header from the request made by Harbor to the token string from the Secret reference. To enable the inclusion of this header, the token string must be configured as the “Auth Header” while configuring a webhook in Harbor.

If the two tokens match, the controller will request a reconciliation for all listed Resources.

Note: This type of Receiver does not support filtering using Events. However, Harbor does support configuring event types for which a webhook will be triggered.

Harbor example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: harbor-receiver
 namespace: default
spec:
 type: harbor
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: webapp

DockerHub

When a Receiver’s .spec.type is set to dockerhub, the controller will respond to an HTTP webhook event payload from DockerHub to the generated .status.webhookPath path.

The controller performs minimal validation of the payload by attempting to unmarshal the JSON request body. If the unmarshalling is successful, the controller will request a reconciliation for all listed Resources.

Note: This type of Receiver does not support filtering using Events.

DockerHub example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: dockerhub-receiver
 namespace: default
spec:
 type: dockerhub
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: webapp

Quay

When a Receiver’s .spec.type is set to quay, the controller will respond to an HTTP Repository Push Notification payload from Quay to the generated .status.webhookPath path.

The controller performs minimal validation of the payload by attempting to unmarshal the JSON request body to the expected format. If the unmarshalling is successful, the controller will request a reconciliation for all listed Resources.

Note: This type of Receiver does not support filtering using Events. In addition, it does not support any “Repository Notification” other than “Repository Push”.

Quay example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: quay-receiver
 namespace: default
spec:
 type: quay
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: webapp

Nexus

When a Receiver’s .spec.type is set to nexus, the controller will respond to an HTTP webhook event payload from Nexus Repository Manager 3 to the generated .status.webhookPath path, while verifying the payload’s integrity and authenticity using HMAC.

The controller validates the payload by comparing the X-Nexus-Webhook-Signature header from the request made by Nexus to the token string from the Secret reference. To enable the inclusion of this header, the token string must be configured as the “Secret Key” while enabling a repository webhook capability.

The controller will calculate the HMAC hash signature for the received request payload using the same token string, and compare it with the one specified in the header. If the two signatures match, the controller will attempt to unmarshal the request body to the expected format. If the unmarshalling is successful, the controller will request a reconciliation for all listed Resources.

Note: This type of Receiver does not support filtering using Events.

Nexus example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: nexus-receiver
 namespace: default
spec:
 type: nexus
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: webapp

GCR

When a Receiver’s .spec.type is set to gcr, the controller will respond to an HTTP webhook event payload from Google Cloud Registry to the generated .status.webhookPath, while verifying the payload is legitimate using JWT.

The controller verifies the request originates from Google by validating the token from the Authorization header. For this to work, authentication must be enabled for the Pub/Sub subscription, refer to the Google Cloud documentation for more information.

When the verification succeeds, the request payload is unmarshalled to the expected format. If this is successful, the controller will request a reconciliation for all listed Resources.

Note: This type of Receiver does not support filtering using Events.

GCR example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: gcr-receiver
 namespace: default
spec:
 type: gcr
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: webapp
 namespace: default

ACR

When a Receiver’s .spec.type is set to acr, the controller will respond to an HTTP webhook event payload, from Azure Container Registry to the generated .status.webhookPath.

Note: This type of Receiver does not support filtering using Events. However, Azure Container Registry does support configuring webhooks to only send events for specific actions.

ACR example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: acr-receiver
 namespace: default
spec:
 type: acr
 secretRef:
 name: webhook-token
 resources:
 - kind: ImageRepository
 name: webapp

CDEvents

When a Receiver’s .spec.type is set to cdevents, the controller will respond to a CDEvent Event Payload. It will verify the CDEvent using the CDEvent Go-SDK.

This type of receiver supports filtering using Events by comparing the type header to the list of events.

CDEvents example

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: cdevents-receiver
 namespace: flux-system
spec:
 type: cdevents
 events:
 - "dev.cdevents.change.merged"
 secretRef:
 name: receiver-token
 resources:
 - kind: GitRepository 
 name: webapp

Events

.spec.events is an optional field to specify a list of webhook payload event types this Receiver should act on. If left empty, no filtering is applied and any (valid) payload is handled.

Note: Support for this field, and the entries in it, is dependent on the Receiver type. See the supported Receiver types section for more information.

Resources

.spec.resources is a required field to specify which Flux Custom Resources should be reconciled when the Receiver’s webhook path is called.

A resource entry contains the following fields:

apiVersion (Optional): The Flux Custom Resource API group and version, such as source.toolkit.fluxcd.io/v1.
kind: The Flux Custom Resource kind, e.g. Bucket, GitRepository, OCIRepository or ImageRepository.
name: The Flux Custom Resource .metadata.name or * (if matchLabels is specified)
namespace (Optional): The Flux Custom Resource .metadata.namespace. When not specified, the Receiver’s .metadata.namespace is used instead.
matchLabels (Optional): Annotate Flux Custom Resources with specific labels. The name field must be set to * when using matchLabels

Reconcile objects by name

To reconcile a single object, set the kind, name and namespace:

resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: podinfo

Reconcile objects by label

To reconcile objects of a particular kind with specific labels:

resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: "*"
 matchLabels:
 app: podinfo

Note: Cross-namespace references can be disabled for security reasons.

Filtering reconciled objects with CEL

To filter the resources that are reconciled you can use Common Expression Language (CEL).

For example, to trigger ImageRepositories on notifications from Google Artifact Registry you can define the following receiver:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: gar-receiver
 namespace: apps
spec:
 type: gcr
 secretRef:
 name: flux-gar-token
 resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: "*"
 matchLabels:
 registry: gar

This will trigger the reconciliation of all ImageRepositories with the label registry: gar.

But if you want to only notify ImageRepository resources that are referenced from the incoming hook you can use CEL to filter the resources.

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: gar-receiver
 namespace: apps
spec:
 type: gcr
 secretRef:
 name: flux-gar-token
 resources:
 - apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository
 name: "*"
 matchLabels:
 registry: gar
 resourceFilter: 'req.tag.contains(res.metadata.name)'

If the body of the incoming hook looks like this:

{
 "action":"INSERT",
 "digest":"us-east1-docker.pkg.dev/my-project/my-repo/hello-world@sha256:6ec128e26cd5...",
 "tag":"us-east1-docker.pkg.dev/my-project/my-repo/hello-world:1.1"
}

This simple example would match ImageRepositories containing the name hello-world.

If you want to do more complex processing:

 resourceFilter: has(res.metadata.annotations) && req.tag.split('/').last().value().split(":").first().value() == res.metadata.annotations['update-image']

This would look for an annotation “update-image” on the resource, and match it to the hello-world part of the tag name.

Note: Currently the resource value in the CEL expression only provides the object metadata, this means you can access things like res.metadata.labels, res.metadata.annotations and res.metadata.name.

Secret reference

.spec.secretRef.name is a required field to specify a name reference to a Secret in the same namespace as the Receiver. The Secret must contain a token key, whose value is a string containing a (random) secret token.

This token is used to salt the generated webhook path, and depending on the Receiver type, to verify the authenticity of a request.

Example:

---
apiVersion: v1
kind: Secret
metadata:
 name: webhook-token
 namespace: default
type: Opaque
stringData:
 token: <random token>

To trigger a reconciliation of the Receiver when changes occur in the referenced Secret, you can set the following label on the Secret:

metadata:
 labels:
 reconcile.fluxcd.io/watch: Enabled

An alternative to labeling every Secret is setting the --watch-configs-label-selector=owner!=helm flag in notification-controller, which allows watching all Secrets except for Helm storage Secrets.

Interval

.spec.interval is an optional field with a default of ten minutes that specifies the time interval at which the controller reconciles the provider with its Secret reference.

Suspend

.spec.suspend is an optional field to suspend the Receiver. When set to true, the controller will stop processing events for this Receiver. When the field is set to false or removed, it will resume.

Working with Receivers

Disabling cross-namespace selectors

On multi-tenant clusters, platform admins can disable cross-namespace references with the --no-cross-namespace-refs=true flag. When this flag is set, Receivers can only refer to Resources in the same namespace as the Receiver object, preventing tenants from triggering reconciliations to another tenant’s resources.

Public Ingress considerations

Considerations should be made when exposing the controller’s webhook-receiver Kubernetes Service to the public internet. Each request to a Receiver webhook path will result in request to the Kubernetes API, as the controller needs to fetch information about the resource. This endpoint may be protected with a token, but this does not defend against a situation where a legitimate webhook caller starts sending large amounts of requests, or the token is somehow leaked. This may result in the controller, as it may get rate limited by the Kubernetes API, degrading its functionality.

It is therefore a good idea to set rate limits on the Ingress which exposes the Kubernetes Service. If you are using ingress-nginx, this can be done by adding annotations.

Triggering a reconcile

To manually tell the notification-controller to reconcile a Receiver outside of the specified interval window, a Receiver can be annotated with reconcile.fluxcd.io/requestedAt: <arbitrary value>. Annotating the resource queues the Receiver for reconciliation if the <arbitrary-value> differs from the last value the controller acted on, as reported in .status.lastHandledReconcileAt.

Using kubectl:

kubectl annotate --field-manager=flux-client-side-apply --overwrite receiver/<receiver-name> reconcile.fluxcd.io/requestedAt="$(date +%s)"

Using flux:

flux reconcile source receiver <receiver-name>

Waiting for `Ready`

When a change is applied, it is possible to wait for the Receiver to reach a ready state using kubectl:

kubectl wait receiver/<receiver-name> --for=condition=ready --timeout=1m

Suspending and resuming

When you find yourself in a situation where you temporarily want to pause the reconciliation of a Receiver and the handling of requests, you can suspend it using the .spec.suspend field.

Suspend a Receiver

In your YAML declaration:

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: <receiver-name>
spec:
 suspend: true

Using kubectl:

kubectl patch receiver <receiver-name> --field-manager=flux-client-side-apply -p '{\"spec\": {\"suspend\" : true }}'

Using flux:

flux suspend receiver <receiver-name>

Resume a Receiver

In your YAML declaration, comment out (or remove) the field:

---
apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: <receiver-name>
spec:
 # suspend: true

Note: Setting the field value to false has the same effect as removing it, but does not allow for “hot patching” using e.g. kubectl while practicing GitOps; as the manually applied patch would be overwritten by the declared state in Git.

Using kubectl:

kubectl patch receiver <receiver-name> --field-manager=flux-client-side-apply -p '{\"spec\" : {\"suspend\" : false }}'

Using flux:

flux resume receiver <receiver-name>

Debugging a Receiver

There are several ways to gather information about a Receiver for debugging purposes.

Describe the Receiver

Describing a Receiver using kubectl describe receiver <receiver-name> displays the latest recorded information for the resource in the Status and Events sections:

...
Status:
...
Status:
 Conditions:
 Last Transition Time: 2022-11-21T12:41:48Z
 Message: Reconciliation in progress
 Observed Generation: 1
 Reason: ProgressingWithRetry
 Status: True
 Type: Reconciling
 Last Transition Time: 2022-11-21T12:41:48Z
 Message: unable to read token from secret 'default/webhook-token' error: Secret "webhook-token" not found
 Observed Generation: 1
 Reason: TokenNotFound
 Status: False
 Type: Ready
 Observed Generation: -1
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Warning Failed 5s (x4 over 16s) notification-controller unable to read token from secret 'default/webhook-token' error: Secret "webhook-token" not found

Trace emitted Events

To view events for specific Receiver(s), kubectl events can be used in combination with --for to list the Events for specific objects. For example, running

kubectl events --for=Receiver/<receiver-name>

lists

LAST SEEN TYPE REASON OBJECT MESSAGE
3m44s Warning Failed receiver/<receiver-name> unable to read token from secret 'default/webhook-token' error: Secret "webhook-token" not found

Receiver Status

Conditions

A Receiver enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be ready, or it can fail during reconciliation.

The Receiver API is compatible with the kstatus specification, and reports the Reconciling condition where applicable.

Ready Receiver

The notification-controller marks a Receiver as ready when it has the following characteristics:

The Receiver’s Secret referenced in .spec.secretRef.name is found on the cluster.
The Receiver’s Secret contains a token key.

When the Receiver is “ready”, the controller sets a Condition with the following attributes in the Alert’s .status.conditions:

type: Ready
status: "True"
reason: Succeeded

Failed Receiver

The notification-controller may get stuck trying to reconcile a Receiver if its secret token can not be found.

When this happens, the controller sets the Ready Condition status to False, and adds a Condition with the following attributes:

type: Reconciling
status: "True"
reason: ProgressingWithRetry

Observed Generation

The notification-controller reports an observed generation in the Receiver’s .status.observedGeneration. The observed generation is the latest .metadata.generation which resulted in a ready state.

Last Handled Reconcile At

The notification-controller reports the last reconcile.fluxcd.io/requestedAt annotation value it acted on in the .status.lastHandledReconcileAt field.

Webhook Path

When a Receiver becomes ready, the controller reports the generated incoming webhook path under .status.webhookPath. The path format is /hook/sha256sum(token+name+namespace).

Flux: Providers

Mon, 01 Jan 0001 00:00:00 +0000

The Provider API defines how events are encoded and where to send them.

Example

The following is an example of how to send alerts to Slack when Flux fails to install or upgrade Flagger.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: slack-bot
 namespace: flagger-system
spec:
 type: slack
 channel: general
 address: https://slack.com/api/chat.postMessage
 secretRef:
 name: slack-bot-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: slack
 namespace: flagger-system
spec:
 summary: "Flagger impacted in us-east-2"
 providerRef:
 name: slack-bot
 eventSeverity: error
 eventSources:
 - kind: HelmRepository
 name: '*'
 - kind: HelmRelease
 name: '*'

In the above example:

A Provider named slack-bot is created, indicated by the Provider.metadata.name field.
An Alert named slack is created, indicated by the Alert.metadata.name field.
The Alert references the slack-bot provider, indicated by the Alert.spec.providerRef field.
The notification-controller starts listening for events sent for all HelmRepositories and HelmReleases in the flagger-system namespace.
When an event with severity error is received, the controller posts a message on Slack containing the summary text and the Helm install or upgrade error.
The controller uses the Slack Bot token from the secret indicated by the Provider.spec.secretRef.name to authenticate with the Slack API.

You can run this example by saving the manifests into slack-alerts.yaml.

First create a secret with the Slack bot token:

kubectl -n flagger-system create secret generic slack-bot-token --from-literal=token=xoxb-YOUR-TOKEN

Apply the resources on the cluster:

kubectl -n flagger-system apply --server-side -f slack-alerts.yaml

Writing a provider spec

As with all other Kubernetes config, a Provider needs apiVersion, kind, and metadata fields. The name of an Alert object must be a valid DNS subdomain name.

A Provider also needs a .spec section.

Type

.spec.type is a required field that specifies which SaaS API to use.

The supported alerting providers are:

Provider	Type
Generic webhook	`generic`
Generic webhook with HMAC	`generic-hmac`
Azure Event Hub	`azureeventhub`
DataDog	`datadog`
Discord	`discord`
GitHub dispatch	`githubdispatch`
Google Chat	`googlechat`
Google Pub/Sub	`googlepubsub`
Grafana	`grafana`
Lark	`lark`
Matrix	`matrix`
Microsoft Teams	`msteams`
Opsgenie	`opsgenie`
PagerDuty	`pagerduty`
Prometheus Alertmanager	`alertmanager`
Rocket	`rocket`
Sentry	`sentry`
Slack	`slack`
Telegram	`telegram`
WebEx	`webex`
NATS	`nats`
Zulip	`zulip`
OTEL	`otel`

Types supporting Git commit status updates

The providers supporting Git commit status updates are:

Provider	Type
Azure DevOps	`azuredevops`
Bitbucket	`bitbucket`
Bitbucket Server/Data Center	`bitbucketserver`
GitHub	`github`
GitLab	`gitlab`
Gitea	`gitea`

Alerting

Generic webhook

When .spec.type is set to generic, the controller will send an HTTP POST request to the provided Address.

The body of the request is a JSON Event object, for example:

{
 "involvedObject": {
 "apiVersion": "kustomize.toolkit.fluxcd.io/v1",
 "kind": "Kustomization",
 "name": "webapp",
 "namespace": "apps",
 "uid": "7d0cdc51-ddcf-4743-b223-83ca5c699632"
 },
 "metadata": {
 "kustomize.toolkit.fluxcd.io/revision": "main/731f7eaddfb6af01cb2173e18f0f75b0ba780ef1"
 },
 "severity":"error",
 "reason": "ValidationFailed",
 "message":"service/apps/webapp validation error: spec.type: Unsupported value: Ingress",
 "reportingController":"kustomize-controller",
 "reportingInstance":"kustomize-controller-7c7b47f5f-8bhrp",
 "timestamp":"2022-10-28T07:26:19Z"
}

Where the involvedObject key contains the metadata from the object triggering the event.

The controller includes a Gotk-Component header in the request, which can be used to identify the component which sent the event, e.g. source-controller or notification-controller.

POST / HTTP/1.1
Host: example.com
Accept-Encoding: gzip
Content-Length: 452
Content-Type: application/json
Gotk-Component: kustomize-controller
User-Agent: Go-http-client/1.1

You can add additional headers to the POST request using a headers key in the referenced Secret.

Generic webhook with HMAC

When .spec.type is set to generic-hmac, the controller will send an HTTP POST request to the provided Address for an Event, while including an X-Signature HTTP header carrying the HMAC of the request body. The inclusion of the header allows the receiver to verify the authenticity and integrity of the request.

The X-Signature header is calculated by generating an HMAC of the request body using the token key from the referenced Secret. The HTTP header value has the following format:

X-Signature: <hash-function>=<hash>

<hash-function> denotes the hash function used to generate the HMAC and currently defaults to sha256, which may change in the future. <hash> is the HMAC of the request body, encoded as a hexadecimal string.

while <hash> is the hex-encoded HMAC value.

The body of the request is a JSON Event object, as described in the Generic webhook section.

HMAC verification example

The following example in Go shows how to verify the authenticity and integrity of a request by using the X-Signature header.

func verifySignature(signature string, payload, key []byte) error {
 sig := strings.Split(signature, "=")

 if len(sig) != 2 {
 return fmt.Errorf("invalid signature value")
 }

 var newF func() hash.Hash
 switch sig[0] {
 case "sha224":
 newF = sha256.New224
 case "sha256":
 newF = sha256.New
 case "sha384":
 newF = sha512.New384
 case "sha512":
 newF = sha512.New
 default:
 return fmt.Errorf("unsupported signature algorithm %q", sig[0])
 }

 mac := hmac.New(newF, key)
 if _, err := mac.Write(payload); err != nil {
 return fmt.Errorf("failed to write payload to HMAC encoder: %w", err)
 }

 sum := fmt.Sprintf("%x", mac.Sum(nil))
 if sum != sig[1] {
 return fmt.Errorf("HMACs do not match: %#v != %#v", sum, sig[1])
 }
 return nil
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
 // Require a X-Signature header
 if len(r.Header["X-Signature"]) == 0 {
 http.Error(w, "missing X-Signature header", http.StatusBadRequest)
 return
 }

 // Read the request body with a limit of 1MB
 lr := io.LimitReader(r.Body, 1<<20)
 body, err := io.ReadAll(lr)
 if err != nil {
 http.Error(w, "failed to read request body", http.StatusBadRequest)
 return
 }

 // Verify signature using the same token as the Secret referenced in
 // Provider
 key := []byte("<token>")
 if err := verifySignature(r.Header.Get("X-Signature"), body, key); err != nil {
 http.Error(w, fmt.Sprintf("failed to verify HMAC signature: %s", err.Error()), http.StatusBadRequest)
 return
 }

 // Do something with the verified request body
 // ...
}

Slack

When .spec.type is set to slack, the controller will send a message for an Event to the provided Slack API Address.

The Event will be formatted into a Slack message using an Attachment, with the metadata attached as fields, and the involved object as author. The severity of the Event is used to set the color of the attachment.

When a Channel is provided, it will be added as a channel field to the API payload. Otherwise, the further configuration of the Address will determine the channel.

When Username is set, this will be added as a username field to the payload, defaulting to the name of the reporting controller.

This Provider type supports the configuration of a proxy URL and/or certificate secret reference.

Slack example

To configure a Provider for Slack, we recommend using a Slack Bot App token which is not attached to a specific Slack account. To obtain a token, please follow Slack’s guide on creating an app.

Once you have obtained a token, create a Secret containing the token key and a slack Provider with the address set to https://slack.com/api/chat.postMessage.

Using this API endpoint, it is possible to send messages to multiple channels by adding the integration to each channel.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: slack
 namespace: default
spec:
 type: slack
 channel: general
 address: https://slack.com/api/chat.postMessage
 secretRef:
 name: slack-token
---
apiVersion: v1
kind: Secret
metadata:
 name: slack-token
 namespace: default
stringData:
 token: xoxb-1234567890-1234567890-1234567890-1234567890

Slack (legacy) example

To configure a Provider for Slack using the legacy incoming webhook API, create a Secret with the address set to https://hooks.slack.com/services/..., and a slack Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: slack
 namespace: default
spec:
 type: slack
 secretRef:
 name: slack-webhook
---
apiVersion: v1
kind: Secret
metadata:
 name: slack-webhook
 namespace: default
stringData:
 address: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

Microsoft Teams

When .spec.type is set to msteams, the controller will send a payload for an Event to the provided Address. The address may be a Microsoft Teams Incoming Webhook Workflow, or the deprecated Office 365 Connector.

Note: If the Address host contains the suffix .webhook.office.com, the controller will imply that the backend is the deprecated Office 365 Connector and is expecting the Event in the connector message format. Otherwise, the controller will format the Event as a Microsoft Adaptive Card message.

In both cases the Event metadata is attached as facts, and the involved object as a summary/title. The severity of the Event is used to set the color of the message.

This Provider type supports the configuration of a proxy URL and/or certificate secret reference, but lacks support for configuring a Channel. This can be configured during the creation of the Incoming Webhook Workflow in Microsoft Teams.

Microsoft Teams example

To configure a Provider for Microsoft Teams, create a Secret with the address set to the webhook URL, and an msteams Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: msteams
 namespace: default
spec:
 type: msteams
 secretRef:
 name: msteams-webhook
---
apiVersion: v1
kind: Secret
metadata:
 name: msteams-webhook
 namespace: default
stringData:
 address: https://prod-xxx.yyy.logic.azure.com:443/workflows/zzz/triggers/manual/paths/invoke?...

DataDog

When .spec.type is set to datadog, the controller will send a payload for an Event to the provided DataDog API Address.

The Event will be formatted into a DataDog Event and sent to the API endpoint of the provided DataDog Address.

This Provider type supports the configuration of a proxy URL and/or certificate secret reference.

The metadata of the Event is included in the DataDog event as extra tags.

DataDog example

To configure a Provider for DataDog, create a Secret with the token set to a DataDog API key (not an application key!) and a datadog Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: datadog
 namespace: default
spec:
 type: datadog
 address: https://api.datadoghq.com # DataDog Site US1
 secretRef:
 name: datadog-secret
---
apiVersion: v1
kind: Secret
metadata:
 name: datadog-secret
 namespace: default
stringData:
 token: <DataDog API Key>
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: datadog-info
 namespace: default
spec:
 eventSeverity: info
 eventSources:
 - kind: HelmRelease
 name: "*"
 providerRef:
 name: datadog
 eventMetadata:
 env: my-k8s-cluster # example of adding a custom `env` tag to the event

Discord

When .spec.type is set to discord, the controller will send a payload for an Event to the provided Discord Address.

The Event will be formatted into a Slack message and send to the /slack endpoint of the provided Discord Address.

Discord example

To configure a Provider for Discord, create a Secret with the address set to the webhook URL, and a discord Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: discord
 namespace: default
spec:
 type: discord
 secretRef:
 name: discord-webhook
---
apiVersion: v1
kind: Secret
metadata:
 name: discord-webhook
 namespace: default
stringData:
 address: "https://discord.com/api/webhooks/..."

Sentry

When .spec.type is set to sentry, the controller will send a payload for an Event to the provided Sentry Address.

Depending on the severity of the Event, the controller will capture a Sentry Eventfor error, or Sentry Transaction Event with a Span for info. The metadata of the Event is included as extra data in the Sentry Event, or as Span tags.

The Provider’s Channel is used to set the environment on the Sentry client.

This Provider type supports the configuration of certificate secret reference.

Sentry example

To configure a Provider for Sentry, create a Secret with the address set to a Sentry DSN, and a sentry Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: sentry
 namespace: default
spec:
 type: sentry
 channel: staging-env
 secretRef:
 name: sentry-webhook
---
apiVersion: v1
kind: Secret
metadata:
 name: sentry-webhook
 namespace: default
stringData:
 address: "https://....@sentry.io/12341234"

Note: The sentry Provider also sends traces for events with the severity info. This can be disabled by setting the spec.eventSeverity field to error on an Alert.

When .spec.type is set to telegram, the controller will send a payload for an Event to the Telegram Bot API.

The Event will be formatted into a message string, with the metadata attached as a list of key-value pairs.

The Provider’s Channel is used to set the receiver of the message. This can be a unique identifier (-1234567890) for the target chat, a unique identifier with the topic identifier (-1234567890:1) for the forum chat, or the username (@username) of the target channel.

This Provider type does not support the configuration of a certificate secret reference.

Note: The Telegram provider always ignores the address field and uses the default Telegram Bot API endpoint (https://api.telegram.org).

Telegram example

To configure a Provider for Telegram, create a Secret with the token obtained from the BotFather, and a telegram Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: telegram
 namespace: default
spec:
 type: telegram
 channel: "@fluxcd" # or "-1557265138" (channel id) or "-1552289257:1" (forum chat id with topic id)
 secretRef:
 name: telegram-token

Matrix

When .spec.type is set to matrix, the controller will send a payload for an Event to the provided Matrix Address.

The Event will be formatted into a message string, with the metadata attached as a list of key-value pairs, and send as a m.room.message text event to the provided Matrix Address.

The Provider’s Channel is used to set the receiver of the message using a room identifier (!1234567890:example.org).

This provider type does support the configuration of TLS certificates.

Matrix example

To configure a Provider for Matrix, create a Secret with the token obtained from the Matrix endpoint, and a matrix Provider with a Secret reference.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: matrix
 namespace: default
spec:
 type: matrix
 address: https://matrix.org
 channel: "!jezptmDwEeLapMLjOc:matrix.org"
 secretRef:
 name: matrix-token

Lark

When .spec.type is set to lark, the controller will send a payload for an Event to the provided Lark Address.

The Event will be formatted into a Lark Message card, with the metadata written to the message string.

This Provider type does not support the configuration of a proxy URL or certificate secret reference.

Lark example

To configure a Provider for Lark, create a Secret with the address obtained from adding a bot to a group, and a lark Provider with a Secret reference.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: lark
 namespace: default
spec:
 type: lark
 secretRef:
 name: lark-webhook
---
apiVersion: v1
kind: Secret
metadata:
 name: lark-webhook
 namespace: default
stringData:
 address: "https://open.larksuite.com/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxx"

Rocket

When .spec.type is set to rocket, the controller will send a payload for an Event to the provided Rocket Address.

The Event will be formatted into a Slack message and send as a payload the provided Rocket Address.

This Provider type does support the configuration of a proxy URL and certificate secret reference.

Rocket example

To configure a Provider for Rocket, create a Secret with the address set to the Rocket webhook URL, and a rocket Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: rocket
 namespace: default
spec:
 type: rocket
 secretRef:
 name: rocket-webhook

Google Chat

When .spec.type is set to googlechat, the controller will send a payload for an Event to the provided Google Chat Address.

The Event will be formatted into a Google Chat card message, with the metadata added as a list of key-value pairs in a widget.

This Provider type does support the configuration of a proxy URL.

Google Chat example

To configure a Provider for Google Chat, create a Secret with the address set to the Google Chat webhook URL, and a googlechat Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: google
 namespace: default
spec:
 type: googlechat
 secretRef:
 name: google-webhook
---
apiVersion: v1
kind: Secret
metadata:
 name: google-webhook
 namespace: default
stringData:
 address: https://chat.googleapis.com/v1/spaces/...

Google Pub/Sub

When .spec.type is set to googlepubsub, the controller will publish the payload of an Event on the Google Pub/Sub Topic ID provided in the Channel field, which must exist in the GCP Project ID provided in the Address field.

This Provider type can optionally use the Secret reference to authenticate on the Google Pub/Sub API by using JSON credentials. The credentials must be specified on the token field of the Secret.

If no JSON credentials are specified, then the automatic authentication methods of the Google libraries will take place, and therefore methods like Workload Identity will be automatically attempted.

The Google identity effectively used for publishing messages must have the required permissions on the Pub/Sub Topic.

You can optionally add attributes to the Pub/Sub message using a headers key in the referenced Secret.

This Provider type does not support the configuration of a proxy URL or certificate secret reference.

Google Pub/Sub with JSON Credentials and Custom Headers Example

To configure a Provider for Google Pub/Sub authenticating with JSON credentials and custom headers, create a Secret with the token set to the necessary JSON credentials, the headers field set to a YAML string-to-string dictionary, and a googlepubsub Provider with the associated Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: googlepubsub-provider
 namespace: desired-namespace
spec:
 type: googlepubsub
 address: <GCP Project ID>
 channel: <Pub/Sub Topic ID>
 secretRef:
 name: googlepubsub-provider-creds
---
apiVersion: v1
kind: Secret
metadata:
 name: googlepubsub-provider-creds
 namespace: desired-namespace
stringData:
 token: <GCP JSON credentials>
 headers: |
 attr1-name: attr1-value
 attr2-name: attr2-value

Google Pub/Sub with Workload Identity Example

To configure a Provider for Google Pub/Sub authenticating with Workload Identity, create a Kubernetes ServiceAccount with the appropriate annotations, and a googlepubsub Provider with the ServiceAccount reference. This method eliminates the need for JSON credentials and enables multi-tenant authentication.

Single tenant approach

This approach uses the notification-controller service account for setting up authentication.

In the default installation, the notification-controller service account is located in the flux-system namespace with name notification-controller.
Configure workload identity with notification-controller as described in the docs here.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: googlepubsub-controller-level
 namespace: flux-system
spec:
 type: googlepubsub
 address: <GCP Project ID>
 channel: <Pub/Sub Topic ID>
 # No serviceAccountName specified - uses controller's identity

Multi-tenant approach

For multi-tenant clusters, set .spec.serviceAccountName of the provider to the service account to be used for authentication. Ensure that the service account has the appropriate annotations for GCP workload identity.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: googlepubsub-tenant-a
 namespace: tenant-a
spec:
 type: googlepubsub
 address: <GCP Project ID>
 channel: <Pub/Sub Topic ID>
 serviceAccountName: tenant-a-pubsub-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: tenant-a-pubsub-sa
 namespace: tenant-a
 annotations:
 # For GKE Workload Identity
 iam.gke.io/gcp-service-account: tenant-a-pubsub@<GCP_PROJECT_ID>.iam.gserviceaccount.com
 # For Workload Identity Federation (non-GKE)
 gcp.auth.fluxcd.io/workload-identity-provider: projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<POOL_ID>/providers/<PROVIDER_ID>

Note: Object-level authentication requires the ObjectLevelWorkloadIdentity feature gate to be enabled:

--feature-gates=ObjectLevelWorkloadIdentity=true

Opsgenie

When .spec.type is set to opsgenie, the controller will send a payload for an Event to the provided Opsgenie Address.

The Event will be formatted into a Opsgenie alert, with the metadata added to the details field as a list of key-value pairs.

This Provider type does support the configuration of a proxy URL and certificate secret reference.

Opsgenie example

To configure a Provider for Opsgenie, create a Secret with the token set to the Opsgenie API token, and a opsgenie Provider with a Secret reference and the address set to https://api.opsgenie.com/v2/alerts.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: opsgenie
 namespace: default
spec:
 type: opsgenie
 address: https://api.opsgenie.com/v2/alerts
 secretRef:
 name: opsgenie-token
---
apiVersion: v1
kind: Secret
metadata:
 name: opsgenie-token
 namespace: default
stringData:
 token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

PagerDuty

When .spec.type is set to pagerduty, the controller will send a payload for an Event to the provided PagerDuty Address.

The Event will be formatted into an Event API v2 payload, triggering or resolving an incident depending on the event’s Severity.

The provider will also send Change Events for info level Severity, which will be displayed in the PagerDuty service’s timeline to track changes.

This Provider type supports the configuration of a proxy URL and certificate secret reference.

The Channel is used to set the routing key to send the event to the appropriate integration.

PagerDuty example

To configure a Provider for Pagerduty, create a pagerduty Provider, set address to the integration URL and channel set to the integration key (also known as a routing key) for your service or event orchestration.

When adding an integration for a service on PagerDuty, it is recommended to use Events API v2 integration.

Note: PagerDuty does not support Change Events when sent to global integrations, such as event orchestration.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: pagerduty
 namespace: default
spec:
 type: pagerduty
 address: https://events.pagerduty.com
 channel: <integrationKey>

If you are sending to a service integration, it is recommended to set your Alert to filter to only those sources you want to trigger an incident for that service. For example:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: my-service-pagerduty
 namespace: default
spec:
 providerRef:
 name: pagerduty
 eventSources:
 - kind: HelmRelease
 name: my-service
 namespace: default

Prometheus Alertmanager

When .spec.type is set to alertmanager, the controller will send a payload for an Event to the provided Prometheus Alertmanager Address.

The Event will be formatted into a firing Prometheus Alertmanager alert, with the metadata added to the labels fields, and the message (and optional .metadata.summary) added as annotations. Event timestamp will be used to set alert start time (.StartsAt).

In addition to the metadata from the Event, the following labels will be added:

Label	Description
alertname	The string Flux followed by the Kind and the reason for the event e.g `FluxKustomizationProgressing`
severity	The severity of the event (`error` or `info`)
reason	The machine readable reason for the objects transition into the current status
kind	The kind of the involved object associated with the event
name	The name of the involved object associated with the event
namespace	The namespace of the involved object associated with the event

Note that due to the way other Flux controllers currently emit events, there’s no way for notification-controller to figure out the time the event ends to set .EndsAt (a reasonable estimate being double the reconciliation interval of the resource involved) that doesn’t involve a Kubernetes API roundtrip. A possible workaround could be setting global.resolve_timeout to an interval large enough for events to reoccur:

global:
 resolve_timeout: 1h

This Provider type does support the configuration of a proxy URL and certificate secret reference.

Prometheus Alertmanager example

To configure a Provider for Prometheus Alertmanager, authentication can be done using either Basic Authentication or a Bearer Token. Both methods are supported, but using authentication is optional based on your setup.

Basic Authentication: Create a Secret with the username and password set to the Basic Auth credentials, and the .spec.address field set to the Prometheus Alertmanager HTTP API URL.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: alertmanager
 namespace: default
spec:
 type: alertmanager
 address: https://<alertmanager-hostport>/api/v2/alerts/
 secretRef:
 name: alertmanager-basic-auth
---
apiVersion: v1
kind: Secret
metadata:
 name: alertmanager-basic-auth
 namespace: default
stringData:
 username: <username>
 password: <password>

Bearer Token Authentication: Create a Secret with the token, and an alertmanager Provider with a Secret reference and the Prometheus Alertmanager HTTP API URL set directly in the .spec.address field.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: alertmanager
 namespace: default
spec:
 type: alertmanager
 address: https://<alertmanager-hostport>/api/v2/alerts/
 secretRef:
 name: alertmanager-token
---
apiVersion: v1
kind: Secret
metadata:
 name: alertmanager-token
 namespace: default
stringData:
 token: <token>

Webex

When .spec.type is set to webex, the controller will send a payload for an Event to the provided Webex Address.

The Event will be formatted into a message string, with the metadata attached as a list of key-value pairs, and send as a Webex message.

The Channel is used to set the ID of the room to send the message to.

This Provider type does support the configuration of a proxy URL and certificate secret reference.

Webex example

To configure a Provider for Webex, create a Secret with the token set to the Webex access token, and a webex Provider with a Secret reference and the address set to https://webexapis.com/v1/messages.

Note: To be able to send messages to a Webex room, the bot needs to be added to the room. Failing to do so will result in 404 errors, logged by the controller.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: webex
 namespace: default
spec:
 type: webex
 address: https://webexapis.com/v1/messages
 channel: <webexSpaceRoomID>
 secretRef:
 name: webex-token
---
apiVersion: v1
kind: Secret
metadata:
 name: webex-token
 namespace: default
stringData:
 token: <bot-token>

NATS

When .spec.type is set to nats, the controller will publish the payload of an Event on the NATS Subject provided in the Channel field, using the server specified in the Address field.

This Provider type can optionally use the Secret reference to authenticate to the NATS server using Username/Password. The credentials must be specified in the username and password fields of the Secret. Alternatively, NATS also supports passing the credentials with the server URL. In this case the address should be provided through a Secret reference.

Additionally if using credentials, the User must have authorization to publish on the Subject provided.

NATS with Username/Password Credentials Example

To configure a Provider for NATS authenticating with Username/Password, create a Secret with the username and password fields set, and add a nats Provider with the associated Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: nats-provider
 namespace: desired-namespace
spec:
 type: nats
 address: <NATS Server URL>
 channel: <Subject>
 secretRef:
 name: nats-provider-creds
---
apiVersion: v1
kind: Secret
metadata:
 name: nats-provider-creds
 namespace: desired-namespace
stringData:
 username: <NATS Username>
 password: <NATS Password>

Zulip

When .spec.type is set to zulip, the controller will send a Zulip message to the provided Address. The address is expected to be an HTTP/S URL pointing to a Zulip server. The POST /api/v1/messages API will be used to send the message. See the API docs here.

The Zulip channel and topic must be specified in the Channel field, separated by a forward slash (/). For example: general/alerts. In this case, the message will be sent to the general channel, under the alerts topic.

The zulip Provider supports the configuration of an HTTP/S proxy, a certificate secret reference for configuring TLS, and a secret reference for HTTP Basic Authentication.

Example of a zulip Provider with HTTP Basic Authentication:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: zulip
 namespace: default
spec:
 type: zulip
 address: https://my-org.zulipchat.com
 channel: my-channel/my-topic
 secretRef:
 name: zulip-basic-auth
---
apiVersion: v1
kind: Secret
metadata:
 name: zulip-basic-auth
 namespace: default
stringData:
 username: my-bot@my-org.zulipchat.com  # the Zulip bot email address
 password: F8KXuAylZOta3L5tjgVm3r1YVruUNGXu # the Zulip bot API key

OTEL

When .spec.type is set to otel, the controller will send distributed tracing data for Events to an OpenTelemetry (OTEL) collector endpoint specified in the Address field.

The controller converts Flux events into OTEL spans with proper trace relationships based on the Flux object hierarchy. Source objects (GitRepository, HelmChart, OCIRepository, Bucket) create root spans, while other objects create child spans within the same trace. Each span includes event metadata as attributes and uses the alert name and namespace as the service identifier.

Spans are correlated using a trace ID generated from the alert UID and revision metadata (depending on the source, the revision could be oci-digest, git commit hash, etc.). Having the following format: <AlertUID>:<revision>. All events from the same alert share the same trace ID, enabling end-to-end visibility across.

This Provider type supports authentication via Secret reference, proxy configuration, and TLS certificates.

Note: HelmRepository events are skipped as they are not considered primary sources for tracing.

OTEL with Basic Authentication Example

To configure otel provider with basic authentication, create a Secret with the username and password fields set, and add a otel provider with the associated Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: otel-provider
 namespace: desired-namespace
spec:
 type: otel
 address: <Server-URL>
 secretRef:
 name: otel-provider-creds
---
apiVersion: v1
kind: Secret
metadata:
 name: otel-provider-creds
 namespace: desired-namespace
stringData:
 username: <OTEL Username>
 password: <OTEL Password>

Address

.spec.address is an optional field that specifies the endpoint where the events are posted. The meaning of endpoint here depends on the specific Provider type being used. For the generic Provider for example this is an HTTP/S address. For other Provider types this could be a project ID or a namespace.

If the address contains sensitive information such as tokens or passwords, it is recommended to store the address in the Kubernetes secret referenced by .spec.secretRef.name. When the referenced Secret contains an address key, the .spec.address value is ignored.

Channel

.spec.channel is an optional field that specifies the channel where the events are posted.

Username

.spec.username is an optional field that specifies the username used to post the events. Can be overwritten with a Secret reference.

Secret reference

.spec.secretRef.name is an optional field to specify a name reference to a Secret in the same namespace as the Provider, containing the authentication credentials for the provider API.

The Kubernetes secret can have any of the following keys:

address - overrides .spec.address
proxy - overrides .spec.proxy (deprecated, use .spec.proxySecretRef instead. Support for this key will be removed in v1)
token - used for authentication
username - overrides .spec.username
password - used for authentication, often in combination with username (or .spec.username)
headers - HTTP headers values included in the POST request

Address example

For providers which embed tokens or other sensitive information in the URL, the incoming webhooks address can be stored in the secret using the address key:

---
apiVersion: v1
kind: Secret
metadata:
 name: my-provider-url
 namespace: default
stringData:
 address: "https://webhook.example.com/token"

Token example

For providers which require token based authentication, the API token can be stored in the secret using the token key:

---
apiVersion: v1
kind: Secret
metadata:
 name: my-provider-auth
 namespace: default
stringData:
 token: "my-api-token"

HTTP headers example

For providers which require specific HTTP headers to be attached to the POST request, the headers can be set in the secret using the headers key:

---
apiVersion: v1
kind: Secret
metadata:
 name: my-provider-headers
 namespace: default
stringData:
 headers: |
 Authorization: my-api-token
 X-Forwarded-Proto: https

Proxy auth example

Some networks need to use an authenticated proxy to access external services. The recommended approach is to use .spec.proxySecretRef with a dedicated Secret:

---
apiVersion: v1
kind: Secret
metadata:
 name: my-provider-proxy
 namespace: default
stringData:
 address: "http://proxy_url:proxy_port"
 username: "proxy_username"
 password: "proxy_password"

Legacy approach (deprecated): The proxy address can also be stored in the main secret to hide parameters like the username and password:

---
apiVersion: v1
kind: Secret
metadata:
 name: my-provider-proxy-legacy
 namespace: default
stringData:
 proxy: "http://username:password@proxy_url:proxy_port"

Certificate secret reference

.spec.certSecretRef is an optional field to specify a name reference to a Secret in the same namespace as the Provider, containing TLS certificates for secure communication. The secret must be of type kubernetes.io/tls or Opaque.

Supported configurations

Mutual TLS (mTLS): Client certificate authentication (provide tls.crt + tls.key, optionally with ca.crt)
CA-only: Server authentication (provide ca.crt only)

Mutual TLS Authentication

Mutual TLS authentication allows for secure client-server communication using client certificates stored in Kubernetes secrets. Both tls.crt and tls.key must be specified together for client certificate authentication. The ca.crt field is optional but required when connecting to servers with self-signed certificates.

Providers supporting client certificate authentication

The following providers support client certificate authentication:

Provider Type	Description
`alertmanager`	Prometheus Alertmanager
`azuredevops`	Azure DevOps
`bitbucket`	Bitbucket
`bitbucketserver`	BitBucket Server/Data Center
`datadog`	DataDog
`discord`	Discord webhooks
`forwarder`	Generic forwarder
`gitea`	Gitea
`github`	GitHub
`githubdispatch`	GitHub Dispatch
`gitlab`	GitLab
`grafana`	Grafana annotations API
`matrix`	Matrix rooms
`msteams`	Microsoft Teams
`opsgenie`	Opsgenie alerts
`pagerduty`	PagerDuty events
`rocket`	Rocket.Chat
`sentry`	Sentry
`slack`	Slack API
`webex`	Webex messages
`zulip`	Zulip API
`otel`	OpenTelemetry Traces

Support for client certificate authentication is being expanded to additional providers over time.

Example: mTLS Configuration

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: my-webhook-mtls
 namespace: default
spec:
 type: generic
 address: https://my-webhook.internal
 certSecretRef:
 name: my-mtls-certs
---
apiVersion: v1
kind: Secret
metadata:
 name: my-mtls-certs
 namespace: default
type: kubernetes.io/tls # or Opaque
stringData:
 tls.crt: |
 -----BEGIN CERTIFICATE-----
 <client certificate>
 -----END CERTIFICATE-----
 tls.key: |
 -----BEGIN PRIVATE KEY-----
 <client private key>
 -----END PRIVATE KEY-----
 ca.crt: |
 -----BEGIN CERTIFICATE-----
 <certificate authority certificate>
 -----END CERTIFICATE-----

CA Certificate Authentication

CA certificate authentication provides server authentication when connecting to HTTPS endpoints with self-signed or custom CA certificates. Only the ca.crt field is required for this configuration.

Example: CA Certificate Configuration

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: my-webhook
 namespace: flagger-system
spec:
 type: generic
 address: https://my-webhook.internal
 certSecretRef:
 name: my-ca-crt
---
apiVersion: v1
kind: Secret
metadata:
 name: my-ca-crt
 namespace: default
type: kubernetes.io/tls # or Opaque
stringData:
 ca.crt: |
 -----BEGIN CERTIFICATE-----
 <certificate authority certificate>
 -----END CERTIFICATE-----

Warning: Support for the caFile key has been deprecated. If you have any Secrets using this key, the controller will log a deprecation warning.

HTTP/S proxy

.spec.proxy is an optional field to specify an HTTP/S proxy address. Warning: This field is deprecated, use .spec.proxySecretRef instead. Support for this field will be removed in v1.

.spec.proxySecretRef is an optional field to specify a name reference to a Secret in the same namespace as the Provider, containing the proxy configuration. The Secret should contain an address key with the HTTP/S address of the proxy server. Optional username and password keys can be provided for proxy authentication.

If the proxy address contains sensitive information such as basic auth credentials, it is recommended to use .spec.proxySecretRef instead of .spec.proxy. When .spec.proxySecretRef is specified, both .spec.proxy and the proxy key from .spec.secretRef are ignored.

Timeout

.spec.timeout is an optional field to specify the timeout for the HTTP/S request sent to the provider endpoint. The value must be in a Go recognized duration string format, e.g. 5m30s for a timeout of five minutes and thirty seconds.

Suspend

.spec.suspend is an optional field to suspend the provider. When set to true, the controller will stop sending events to this provider. When the field is set to false or removed, it will resume.

Working with Providers

Grafana

To send notifications to Grafana annotations API, enable the annotations on a Dashboard like so:

Annotations > Query > Enable Match any
Annotations > Query > Tags (Add Tag: flux)

If Grafana has authentication configured, create a Kubernetes Secret with the API token:

kubectl create secret generic grafana-token \
--from-literal=token=<grafana-api-key> \

Grafana can also use basic authorization to authenticate the requests, if both the token and the username/password are set in the secret, then token takes precedence over`basic auth:

kubectl create secret generic grafana-token \
--from-literal=username=<your-grafana-username> \
--from-literal=password=<your-grafana-password>

Create a provider of type grafana and reference the grafana-token secret:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: grafana
 namespace: default
spec:
 type: grafana
 address: https://<grafana-url>/api/annotations
 secretRef:
 name: grafana-token

Besides the tag flux and the tag containing the reporting controller (e.g. source-controller), the event metadata is also included as tags of the form ${metadataKey}: ${metadataValue}, and the tags kind: ${event.InvolvedObject.Kind}, name: ${event.InvolvedObject.Name} and namespace: ${event.InvolvedObject.Namespace} are also included.

GitHub dispatch

The githubdispatch provider generates GitHub events of type repository_dispatch for the selected repository. The repository_dispatch events can be used to trigger GitHub Actions workflow.

The request includes the event_type and client_payload fields:

event_type is generated from the involved object in the format {Kind}/{Name}.{Namespace}.
client_payload contains the Flux event.

Setting up the GitHub dispatch provider

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: github-dispatch
 namespace: flux-system
spec:
 type: githubdispatch
 address: https://github.com/stefanprodan/podinfo
 secretRef:
 name: auth-secret

The address is the address of your repository where you want to send webhooks to trigger GitHub workflows.

GitHub uses personal access tokens or GitHub app for authentication with its API:

GitHub personal access token

The provider requires a secret in the same format, with the personal access token as the value for the token key:

---
apiVersion: v1
kind: Secret
metadata:
 name: auth-secret
 namespace: default
stringData:
 token: <personal-access-tokens>

GitHub App

To use Github App authentication, make sure the GitHub App is registered and installed with the necessary permissions and the GitHub app secret is created as described here.

NOTE: (For GitHub Enterprise Server) If the GitHub Server uses a private CA, the CA certificate can be referenced either via .spec.certSecretRef as described here or the CA certificate can be added in the GitHub App secret referenced via .spec.secretRef. If the .spec.secretRef contains tls.crt, tls.key then mutual TLS configuration will be automatically enabled. Omit these keys if the GitHub server does not support mutual TLS. If both secret references are specified, then the CA specified in .spec.certSecretRef takes precedence over the CA specified in the GitHub App secret.

Setting up a GitHub workflow

To trigger a GitHub Actions workflow when a Flux Kustomization finishes reconciling, you need to set the event type for the repository_dispatch trigger to match the Flux object ID:

name: test-github-dispatch-provider
on:
 repository_dispatch:
 types: [Kustomization/app1.apps]

Assuming that we deploy all Flux kustomization resources in the same namespace, it will be useful to have a unique kustomization resource name for each application. This will allow you to use only event_type to trigger tests for the exact application.

Let’s say we have the following kustomize overlays structure for two applications:

apps/
├── app1
│   └── overlays
│   ├── production
│   └── staging
└── app2
 └── overlays
 ├── production
 └── staging

You can then create a Flux Kustomization resource for the app to have unique event_type per app. The kustomization manifest for app1/staging:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: app1
 namespace: apps
spec:
 path: "./app1/staging"

You would also like to know from the notification which environment the event is coming from. You can add the spec.eventMetadata field to the Flux Alert and mention the relevant environment info:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: github-dispatch
 namespace: flux-system
spec:
 eventMetadata:
 env: "staging"
 region: "us-west-2"
 providerRef:
 name: github-dispatch
 eventSeverity: info
 eventSources:
 - kind: Kustomization
 name: app1

Now you can trigger the tests in the GitHub workflow for app1 in a staging cluster when the app1 resources defined in ./app1/staging/ are reconciled by Flux:

name: test-github-dispatch-provider
on:
 repository_dispatch:
 types: [Kustomization/app1.apps]
jobs:
 run-tests-staging:
 if: |
 github.event.client_payload.metadata.env == 'staging' &&
 github.event.client_payload.severity == 'info'
 runs-on: ubuntu-latest
 steps:
 - name: Run tests
 run: echo "running tests.."

Azure Event Hub

The Azure Event Hub provider supports the following authentication methods,

Managed Identity
SAS based.

Managed Identity

Managed identity authentication can be setup using Azure Workload identity.

Pre-requisites

Ensure Workload Identity is properly set up on your cluster.

Configure workload identity

Create a managed identity to access Azure Event Hub.
Grant the managed identity the necessary permissions to send events to Azure Event hub as described here.
Establish a federated identity credential between the managed identity and the service account to be used for authentication. Ensure the federated credential uses the correct namespace and name of the service account. For more details, please refer to this guide.

Single tenant approach

This approach uses the notification-controller service account for setting up authentication.

In the default installation, the notification-controller service account is located in the flux-system namespace with name notification-controller.
Configure workload identity with notification-controller as described in the docs here.

Multi-tenant approach

For multi-tenant clusters, set .spec.serviceAccountName of the provider to the service account to be used for authentication. Ensure that the service account has the annotations for the client-id and tenant-id of the managed identity.

For a complete guide on how to set up authentication for an Azure Event Hub, see the integration docs.

SAS based auth

When using SAS auth, we only use the address field in the secret.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: azure
 namespace: default
spec:
 type: azureeventhub
 secretRef:
 name: azure-webhook
---
apiVersion: v1
kind: Secret
metadata:
 name: azure-webhook
 namespace: default
stringData:
 address: <SAS-URL>

Assuming that you have created the Azure event hub and namespace you should be able to use a similar command to get your connection string. This will give you the default Root SAS, which is NOT supposed to be used in production.

az eventhubs namespace authorization-rule keys list --resource-group <rg-name> --namespace-name <namespace-name> --name RootManageSharedAccessKey -o tsv --query primaryConnectionString
# The output should look something like this:
Endpoint=sb://fluxv2.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=yoursaskeygeneatedbyazure;EntityPath=youreventhub

To create the needed secret:

kubectl create secret generic azure-webhook \
--from-literal=address="Endpoint=sb://fluxv2.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=yoursaskeygeneatedbyazure"

Git Commit Status Updates

The notification-controller can mark Git commits as reconciled by posting Flux Kustomization events to the origin repository using Git SaaS providers APIs.

Example

The following is an example of how to update the Git commit status for the GitHub repository where Flux was bootstrapped with flux bootstrap github --owner=my-gh-org --repository=my-gh-repo.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: github-status
 namespace: flux-system
spec:
 type: github
 address: https://github.com/my-gh-org/my-gh-repo
 secretRef:
 name: github-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: github-status
 namespace: flux-system
spec:
 providerRef:
 name: github-status
 eventSources:
 - kind: Kustomization
 name: flux-system

GitHub

When .spec.type is set to github, the referenced secret can contain a personal access token OR github app details.

Personal Access Token

To use personal access tokens, the secret must contain a key called token with the value set to a GitHub personal access token. The token must have permissions to update the commit status for the GitHub repository specified in .spec.address.

You can create the secret with kubectl like this:

kubectl create secret generic github-token --from-literal=token=<GITHUB-TOKEN>

GitHub App

To use Github App authentication, make sure the GitHub App is registered and installed with the necessary permissions to update the commit status and the github app secret is created as described here.

GitLab

When .spec.type is set to gitlab, the referenced secret must contain a key called token with the value set to a GitLab personal access token. If available group and project access tokens can also be used.

The token must have permissions to update the commit status for the GitLab repository specified in .spec.address (grant it the api scope).

You can create the secret with kubectl like this:

kubectl create secret generic gitlab-token --from-literal=token=<GITLAB-TOKEN>

For gitlab.com and current self-hosted gitlab installations that support only the Gitlab v4 API you need to use the project ID in the address instead of the project name. Use an address like https://gitlab.com/1234 (with 1234 being the project ID). You can find out the ID by opening the project in the browser and clicking on the three dot button in the top right corner. The menu that opens shows the project ID.

Gitea

When .spec.type is set to gitea, the referenced secret must contain a key called token with the value set to a Gitea token.

The token must have at least the write:repository permission for the provider to update the commit status for the Gitea repository specified in .spec.address.

Gitea 1.20.0 & 1.20.1

Due to a bug in Gitea 1.20.0 and 1.20.1, these versions require the additional read:misc scope to be applied to the token.

You can create the secret with kubectl like this:

kubectl create secret generic gitea-token --from-literal=token=<GITEA-TOKEN>

BitBucket

When .spec.type is set to bitbucket, the referenced secret must contain a key called token with the value set to a BitBucket username and an app password in the format <username>:<app-password>.

The app password must have Repositories (Read/Write) permission for the BitBucket repository specified in .spec.address.

You can create the secret with kubectl like this:

kubectl create secret generic bitbucket-token --from-literal=token=<username>:<app-password>

BitBucket Server/Data Center

When .spec.type is set to bitbucketserver, the following auth methods are available:

Basic Authentication (username/password)
HTTP access tokens

For Basic Authentication, the referenced secret must contain a password field. The username field can either come from the .spec.username field of the Provider or can be defined in the referenced secret.

You can create the secret with kubectl like this:

kubectl create secret generic bb-server-username-password --from-literal=username=<username> --from-literal=password=<password>

For HTTP access tokens, the secret can be created with kubectl like this:

kubectl create secret generic bb-server-token --from-literal=token=<token>

The HTTP access token must have Repositories (Read/Write) permission for the repository specified in .spec.address.

NOTE: Please provide HTTPS clone URL in the address field of this provider. SSH URLs are not supported by this provider type.

Azure DevOps

The following authentication methods can be used when .spec.type is set to azuredevops.

Managed Identity

Managed Identity authentication can be setup using Azure Workload identity.

Pre-requisites

Ensure Workload Identity is properly set up on your cluster.

Configure Workload Identity

Create a Managed Identity and grant the necessary permissions to list/update commit status.
Establish a federated identity credential between the managed identity and the service account to be used for authentication. Ensure the federated credential uses the correct namespace and name of the service account. For more details, please refer to this guide. The service account used for authentication can be single-tenant or multi-tenant. For a complete guide on how to set up authentication, see the integration docs.

Azure DevOps with Workload Identity Example

Single tenant approach

This approach uses the notification-controller service account for setting up authentication.

In the default installation, the notification-controller service account is located in the flux-system namespace with name notification-controller.
Configure workload identity with notification-controller as described in the docs here.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: azuredevops-controller-level
 namespace: flux-system
spec:
 type: azuredevops
 address: https://dev.azure.com/<organization>/<project>/_git/<repository>
 # No serviceAccountName specified - uses controller's identity

Multi-tenant approach

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: azuredevops-tenant-a
 namespace: tenant-a
spec:
 type: azuredevops
 address: https://dev.azure.com/<organization>/<project>/_git/<repository>
 serviceAccountName: tenant-a-devops-sa
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: tenant-a-devops-sa
 namespace: tenant-a
 annotations:
 azure.workload.identity/client-id: <client-id-of-managed-identity>
 azure.workload.identity/tenant-id: <tenant-id>

Note: Object-level authentication requires the ObjectLevelWorkloadIdentity feature gate to be enabled:

--feature-gates=ObjectLevelWorkloadIdentity=true

PAT

The .spec.secretRef must contain a key called token with the value set to a Azure DevOps personal access token.

The token must have permissions to update the commit status for the Azure DevOps repository specified in .spec.address.

You can create the secret with kubectl like this:

kubectl create secret generic azuredevops-token --from-literal=token=<AZURE-TOKEN>

Custom Commit Status Messages

Git providers supporting commit status updates can use a CEL expression to build a custom commit status message by setting the optional field spec.commitStatusExpr:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: github-status
 namespace: flux-system
spec:
 type: github
 address: https://github.com/my-gh-org/my-gh-repo
 secretRef:
 name: github-token
 commitStatusExpr: "(event.involvedObject.kind + '/' + event.involvedObject.name + '/' + event.metadata.foo + '/' + provider.metadata.uid.split('-').first().value()).lowerAscii()"

The CEL expression can access the following variables:

event: The Flux event object containing metadata about the reconciliation
provider: The Provider object
alert: The Alert object

If the spec.commitStatusExpr field is not specified, the notification-controller will use a default commit status message based on the involved object kind, name, and a truncated provider UID to generate a commit status (e.g. kustomization/gitops-system/0c9c2e41).

A useful tool for building and testing CEL expressions is the CEL Playground.

Flux: API Reference

Mon, 01 Jan 0001 00:00:00 +0000

Flux – Notification Controller

Flux: Controller Options

Flags

Feature Gates

Flux: Alerts

Example

Writing an Alert spec

Summary

Provider reference

Event sources

Select objects by name

Select all objects in a namespace

Select objects by label

Disable cross-namespace selectors

Event metadata

Example

Event metadata from object annotations

Example

Event severity

Event exclusion

Example

Event inclusion

Example

Suspend

Flux: Events

Example

Event structure

Rate limiting

Flux: Receivers

Example

Writing a Receiver spec

Type

Supported Receiver types

Generic

Generic example

Generic HMAC

Generic HMAC example

HMAC signature generation example

GitHub

GitHub example

Gitea

GitLab

GitLab example

Bitbucket Server

Bitbucket Server example

Harbor

Harbor example

DockerHub

DockerHub example

Quay

Quay example

Nexus

Nexus example

GCR

GCR example

ACR

ACR example

CDEvents

CDEvents example

Events

Resources

Reconcile objects by name

Reconcile objects by label

Filtering reconciled objects with CEL

Secret reference

Interval

Suspend

Working with Receivers

Disabling cross-namespace selectors

Public Ingress considerations

Triggering a reconcile

Waiting for Ready

Suspending and resuming

Suspend a Receiver

Resume a Receiver

Debugging a Receiver

Describe the Receiver

Trace emitted Events

Receiver Status

Conditions

Waiting for `Ready`