Flux – Flux monitoring

Flux: Flux alerts

Mon, 01 Jan 0001 00:00:00 +0000

When operating a cluster, different teams may wish to receive notifications about the status of their GitOps pipelines. For example, the on-call team would receive alerts about reconciliation failures in the cluster, while the dev team may wish to be alerted when a new version of an app was deployed and if the deployment is healthy.

Prerequisites

To follow this guide you’ll need a Kubernetes cluster bootstrap with Flux. Please see the get started guide or the installation guide.

The Flux controllers emit Kubernetes events whenever a resource status changes. You can use the notification-controller to forward these events to Slack, Microsoft Teams, Discord and others. The notification controller is part of the default Flux installation.

Define a provider

First create a secret with your Slack bot token:

kubectl -n flagger-system create secret generic slack-bot-token \
--from-literal=token=xoxb-YOUR-TOKEN

Create a notification provider for Slack by referencing the above secret:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: slack-bot
 namespace: flagger-system
spec:
 type: slack
 channel: general
 address: https://slack.com/api/chat.postMessage
 secretRef:
 name: slack-bot-token

Providers

Flux supports various providers such as Discord, PagerDuty, Teams, Telegram, Sentry and many others. For a complete list please see the Provider .spec.type documentation.

Define an alert

Create an alert definition for all repositories and kustomizations:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: on-call-webapp
 namespace: flux-system
spec:
 summary: "cluster addons"
 eventMetadata:
 env: "production"
 cluster: "my-cluster"
 region: "us-east-2"
 providerRef:
 name: slack-bot
 eventSeverity: info
 eventSources:
 - kind: GitRepository
 name: '*'
 - kind: Kustomization
 name: '*'

Apply the above files or commit them to the bootstrap repository.

To verify that the alert has been acknowledged by the notification controller do:

flux get alerts

Multiple alerts can be used to send notifications to different channels or Slack organizations.

The event severity can be set to info or error. When the severity is set to error, the kustomize controller will alert on any error encountered during the reconciliation process. This includes kustomize build and validation errors, apply errors and health check failures.

When the verbosity is set to info, the controller will alert if:

a Kubernetes object was created, updated or deleted
health checks are passing
a dependency is delaying the execution
an error occurs

Git commit status

The GitHub, GitLab, Gitea, Bitbucket, and Azure DevOps providers are slightly different to the other providers. Instead of a stateless stream of events, the Git notification providers will link the event with accompanying Git commit which triggered the event. The linking is done by updating the commit status of a specific commit.

In GitHub the commit status set by notification-controller will result in a green checkmark or red cross next to the commit hash. Clicking the icon will show more detailed information about the status.

Receiving an event in the form of a commit status rather than a message in a chat conversation has the benefit that it closes the deployment loop giving quick and visible feedback if a commit has reconciled and if it succeeded. This means that a deployment will work in a similar manner that people are used to with “traditional” push based CD pipelines. Additionally the status can be fetched from the git providers API for a specific commit. Allowing for custom automation tools that can automatically promote, commit to a new directory, after receiving a successful commit status. This can all be done without requiring any access to the Kubernetes cluster.

As stated before the provider works by referencing the same git repository as the Kustomization controller does. When a new commit is pushed to the repository, source-controller will sync the commit, triggering the kustomize-controller to reconcile the new commit. After this is done the kustomize-controller sends an event to the notification-controller with the result and the commit hash it reconciled. Then notification-controller can update the correct commit and repository when receiving the event.

Limitations

The git notification providers require that a commit hash present in the metadata of the event. Therefore, the providers will only work with Kustomization as an event source, as it is the only resource which includes this data.

First follow the get started guide if you do not have a Kubernetes cluster with Flux installed in it. You will need a authentication token to communicate with the API. The authentication method depends on the git provider used, refer to the Provider CRD for details about how to get the correct token. The guide will use GitHub, but the other providers will work in a very similar manner. The token will need to have write access to the repository it is going to update the commit status in. Store the generated token in a Secret with the following data format in the cluster.

apiVersion: v1
kind: Secret
metadata:
 name: github
 namespace: flux-system
data:
 token: <token>

When sending notification events the kustomization-controller will include the commit hash related to the event. Note that the commit hash in the event does not come from the git repository the Kustomization resource comes from but rather the kustomization source ref. This mean that commit status notifications will not work if the manifests comes from a repository which the API token is not allowed to write to.

Copy the manifest content in the “ kustomize” directory into the directory “./clusters/my-cluster/podinfo” in your fleet-infra repository. Make sure that you also add the namespace podinfo.

apiVersion: v1
kind: Namespace
metadata:
 name: podinfo

Then create a Kustomization to deploy podinfo.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: podinfo
 namespace: flux-system
spec:
 interval: 5m
 targetNamespace: podinfo
 path: ./clusters/my-cluster/podinfo
 prune: true
 sourceRef:
 kind: GitRepository
 name: flux-system
 healthChecks:
 - apiVersion: apps/v1
 kind: Deployment
 name: podinfo
 namespace: podinfo
 timeout: 1m

Creating a git provider is very similar to creating other types of providers. The only caveat being that the provider address needs to point to the same git repository as the event source originates from.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: flux-system
 namespace: flux-system
spec:
 type: github
 address: https://github.com/<username>/fleet-infra
 secretRef:
 name: github
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: podinfo
 namespace: flux-system
spec:
 providerRef:
 name: flux-system
 eventSeverity: info
 eventSources:
 - kind: Kustomization
 name: podinfo
 namespace: flux-system

By now the fleet-infra repository should have a similar directory structure.

fleet-infra
└── clusters/
└── my-cluster/
├── flux-system/
│ ├── gotk-components.yaml
│ ├── gotk-sync.yaml
│ └── kustomization.yaml
├── podinfo/
│ ├── namespace.yaml
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── service.yaml
│ └── kustomization.yaml
├── podinfo-kustomization.yaml
└── podinfo-notification.yaml

If podinfo is deployed and the health checks pass you should get a successful status in your forked podinfo repository.

If everything is setup correctly there should now be a green check-mark next to the latest commit. Clicking the check-mark should show a detailed view.

GitHub	GitLab

Generate error

A deployment failure can be forced by setting an invalid image tag in the podinfo deployment.

apiVersion: apps/v1
kind: Deployment
spec:
 template:
 spec:
 containers:
 - name: podinfod
 image: ghcr.io/stefanprodan/podinfo:fake

After the commit has been reconciled it should return a failed commit status. This is where the health check in the Kustomization comes into play together with the timeout. The health check is used to asses the health of the Kustomization. A failed commit status will not be sent until the health check timeout. Setting a lower timeout will give feedback faster, but may sometimes not allow enough time for a new application to deploy.

GitHub	GitLab

Status changes

The provider will continuously receive events as they happen, and multiple events may be received for the same commit hash. The git providers are configured to only update the status if the status has changed. This is to avoid spamming the commit status history with the same status over and over again.

There is an aspect of statefullness that needs to be considered, compared to the other notification providers, as the events are stored by the git provider. This means that the status of a commit can change over time. Initially a deployment may be healthy, resulting in a successful status. Down the line the application, and the health check, may start failing due to the amount of traffic it receives or external dependencies no longer being available. The change in the health check would cause the status to go from successful to failed. It is important to keep this in mind when building any automation tools that deals with the status, and consider the fact that receiving a successful status once does not mean it will always be successful.

Grafana annotations

To display the Flux notifications on Grafana dashboards you can configure Flux to push events to Grafana annotations API:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: grafana
 namespace: monitoring
spec:
 providerRef:
 name: grafana
 eventSeverity: info
 eventSources:
 - kind: GitRepository
 name: '*'
 namespace: flux-system
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: grafana
 namespace: monitoring
spec:
 type: grafana
 address: "http://kube-prometheus-stack-grafana.monitoring/api/annotations"
 secretRef:
 name: grafana-auth

For more details on how to integrate Flux with Grafana API please see the Grafana provider documentation.

Flux: Flux Prometheus metrics

Mon, 01 Jan 0001 00:00:00 +0000

Flux has native support for Prometheus metrics to provide insights into the state of the Flux components. These can be used to set up monitoring for the Flux controllers. In addition, Flux Custom Resource metrics can also be collected leveraging tools like kube-state-metrics. This document provides information about Flux metrics that can be used to set up monitoring, with some examples.

The fluxcd/flux2-monitoring-example repository provides a ready-made example setup to get started with monitoring Flux. It is recommended to set up the monitoring example before continuing with this document to follow along. Before getting into the monitoring setup, the following sections will describe the kinds of metrics that can be collected for Flux.

Controller metrics

The default installation of Flux controllers export Prometheus metrics at port 8080 in the standard /metrics path. These metrics are about the inner workings of the controllers.

Flux resource reconciliation duration metrics:

gotk_reconcile_duration_seconds_bucket{kind, name, namespace, le}
gotk_reconcile_duration_seconds_sum{kind, name, namespace}
gotk_reconcile_duration_seconds_count{kind, name, namespace}

Cache event metrics:

gotk_cache_events_total{event_type, name, namespace}

Token cache metrics:

gotk_token_cache_events_total{event_type, kind, name, namespace, operation}
gotk_token_cache_requests_total{status}
gotk_token_cache_evictions_total
gotk_token_cached_items

Controller CPU and memory usage:

process_cpu_seconds_total{namespace, pod}
container_memory_working_set_bytes{namespace, pod}

Kubernetes API usage:

rest_client_requests_total{namespace, pod}

Controller runtime:

workqueue_longest_running_processor_seconds{name}
controller_runtime_reconcile_total{controller, result}

In addition, many other Go runtime and controller-runtime metrics are also exported.

Resource metrics

Metrics for the Flux custom resources can be used to monitor the deployment of workloads. Since the use case for these metrics may vary depending on the needs, it’s hard to decide which fields of the resources would be useful to the users. Hence, these metrics are not exported by the Flux controllers themselves but can be collected and exported by using other tools that can read the custom resource state from the kube-apiserver. One such tool is kube-state-metrics (KSM). KSM is also deployed as part of kube-prometheus-stack and is used to export the metrics of kubernetes core resources. It can be configured to also collect custom resource metrics. The monitoring setup in flux2-monitoring-example uses KSM to collect and export Flux custom resource metrics.

In the example monitoring setup, the metric gotk_resource_info provides information about the current state of Flux resources.

gotk_resource_info{customresource_group, customresource_kind, customresource_version, exported_namespace, name, ready, suspended, ...}

customresource_group is the API group of the resource, for example source.toolkit.fluxcd.io for the Flux source API.
customresource_kind is the kind of the resource, for example a GitRepository source.
customresource_version is the API version of the resource, for example v1.
exported_namespace is the namespace of the resource.
name is the name of the resource.
ready shows the readiness of the resource.
suspended shows if the resource’s reconciliation is suspended.

These are some of the common labels that are present in metrics for all the kinds of resources. In addition, there are a few resource kind specific labels. See the following table for a list of labels associated with specific resource kind.

Resource Kind	Labels
Kustomization	`revision`, `source_name`
HelmRelease	`revision`, `chart_name`, `chart_app_version`, `chart_source_name`, `chart_ref_name`
GitRepository	`revision`, `url`
Bucket	`revision`, `endpoint`, `bucket_name`
HelmRepository	`revision`, `url`
HelmChart	`revision`, `chart_name`, `chart_version`
OCIRepository	`revision`, `url`
Receiver	`webhook_path`
ImageRepository	`image`
ImagePolicy	`source_name`
ImageUpdateAutomation	`source_name`

Note: The above metric may have extra labels after being collected in Prometheus. This may be due to the default Prometheus scrape configuration used by kube-prometheus-stack. Since they are about the kube-state-metrics service and not about Flux itself, they can be ignored.

gotk_resource_info is an example of a metric used to collect information about the Flux resources. This metric can be customized to add more labels, or more such metrics can also be created by changing the kube-state-metrics custom resource state configuration. Please see Flux custom Prometheus metrics for details about them.

Monitoring setup

In the monitoring example repository, the monitoring configurations can be found in the monitoring/ directory. monitoring/controllers/ directory contains the configurations for deploying kube-prometheus-stack and loki-stack. We’ll discuss kube-prometheus-stack below. For Flux log collection using Loki, refer to the Flux logs docs.

The configuration in the monitoring/controllers/kube-prometheus-stack/ directory creates a HelmRepository of type OCI for the prometheus-community helm charts and a HelmRelease to deploy the kube-prometheus-stack chart in the monitoring namespace. This installs all the monitoring components in the monitoring namespace. Please see the values used for the chart deployment and modify them accordingly.

The chart values used for configuring kube-state-metrics are in the file kube-state-metrics-config.yaml, as seen in the kustomization.yaml, which uses a kustomize ConfigMap generator to put the configurations in a ConfigMap and use the chart values from the ConfigMap. These values are merged with the inline chart values in the HelmRelease. Kube-state-metrics values are in a separate file to make it easier to customize the metrics it collects; refer to the Flux custom Prometheus metrics docs to see how they are used. Once deployed with these values, the kube-state-metrics starts collecting and exporting the Flux resource metrics.

To configure Prometheus to scrape Flux controller metrics, a PodMonitor is used that selects all the Flux controller Pods and sets the metrics endpoint to the http-prom port. Once created, the prometheus-operator will automatically configure Prometheus to scrape the Flux controller metrics.

Flux Grafana dashboards

The example monitoring setup provides two example Grafana dashboards in monitoring/configs/dashboards that use the Flux controller and resource metrics. The Flux Cluster Stats dashboard shows the overall state of the Flux Sources and Cluster Reconcilers. The Flux Control Plane dashboard shows the statistics of the various components that constitute the Flux Control Plane and their operational metrics.

Control plane dashboard:

Cluster reconciliation dashboard:

More custom metrics can be created and used in the dashboards for monitoring Flux.

Flux: Flux custom Prometheus metrics

Mon, 01 Jan 0001 00:00:00 +0000

By default, the standard installation of Flux exports a specific set of metrics about the controllers and their inner workings that may not serve the needs of all the users. Some of these metrics are common across all the Flux controllers, and some are very specific to a few controllers. It’s not feasible to add all the possible informational labels to these metrics, as that may increase the cardinality of the metrics. Since these metrics are about the operations of Flux, it may not be much of a benefit to add custom labels to these metrics, as these are more useful for the people administering and maintaining Flux. Most of the time, the users of Flux who interact with Flux through the Flux custom resources want to know about the resources they work with. For example, the state of GitRepositories and their branches or tag references. These metrics can be scraped by using kube-state-metrics (KSM), which is part of the kube-prometheus-stack. KSM can be configured to add custom labels to the resource metrics, for example, some value from the status of a resource or some arbitrary value like a team name, department name, etc.

Set up kube-state-metrics

Kube-state-metrics can be installed along with the whole monitoring stack using kube-prometheus-stack. The fluxcd/flux2-monitoring-example repository contains example configurations for deploying and configuring kube-prometheus-stack to monitor Flux. These configurations will be discussed in detail in the following sections to show how they can be customized.

The Kube-prometheus-stack Helm chart is used to install the monitoring stack. The kube-state-metrics related configuration in the chart values exists in a separate file called kube-state-metrics-config.yaml. It configures KSM to run in custom-resource-state-only mode. In this state, KSM will not collect metrics for any of the Kubernetes core resources. The rbac section provides KSM access to list and watch Flux custom resources. If image-reflector-controller and image-automation-controllers are not used, the API group (image.toolkit.fluxcd.io) and resources (imagerepositories, imagepolicies, imageupdateautomations) can be removed. The customResourceState section configures how the Flux metrics are composed.

Once deployed, KSM will start collecting the Flux resource metrics from the kube-apiserver and exporting them as configured.

Adding custom metrics

The example customResourceState values used in the above setup add a metric called gotk_resource_info with labels name, exported_namespace, suspended, ready, etc.

- name: "resource_info"
 help: "The current state of a GitOps Toolkit resource."
 each:
 type: Info
 info:
 labelsFromPath:
 name: [metadata, name]
 labelsFromPath:
 exported_namespace: [metadata, namespace]
 suspended: [spec, suspend]
 ready: [status, conditions, "[type=Ready]", status]
 ...

This provides the current state of the Flux resources. It can be used to monitor the readiness of Flux resources. This can be edited to add more labels; more about that in the next section.

Similarly, more custom metrics can be added by appending them to the metrics list. For example, to create a metric about the HelmRelease last applied chart and app version, append the HelmRelease resource metrics section:

...
customResourceState:
 config:
 spec:
 resources:
 - groupVersionKind:
 group: helm.toolkit.fluxcd.io
 version: "v2"
 kind: HelmRelease
 metricNamePrefix: gotk
 metrics:
 - name: "resource_info"
 help: "The current state of a GitOps Toolkit resource."
 each:
 type: Info
 info:
 labelsFromPath:
 name: [metadata, name]
 labelsFromPath:
 exported_namespace: [metadata, namespace]
 suspended: [spec, suspend]
 ready: [status, conditions, "[type=Ready]", status]
 - name: "helmrelease_version_info"
 help: "The version information of helm release resource."
 each:
 type: Info
 info:
 labelsFromPath:
 chart_version: [status, history, "0", chartVersion]
 app_version: [ status, history, "0", appVersion ]
 labelsFromPath:
 name: [metadata, name]
 exported_namespace: [metadata, namespace]
 chart_name: [spec, chart, spec, chart]
...

In the above, gotk_resource_info and gotk_helmrelease_version_info metrics will be exported for HelmReleases.

# HELP gotk_resource_info The current state of a GitOps Toolkit resource.
# TYPE gotk_resource_info info
gotk_resource_info{customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="monitoring",name="kube-prometheus-stack",ready="True"} 1
# HELP gotk_helmrelease_version_info The version information of helm release resource.
# TYPE gotk_helmrelease_version_info info
gotk_helmrelease_version_info{chart_mame="kube-prometheus-stack",chart_version="61.3.2",app_version="v0.75.1",customresource_group="helm.toolkit.fluxcd.io",customresource_kind="HelmRelease",customresource_version="v2",exported_namespace="monitoring",name="kube-prometheus-stack"} 1

Adding custom metric labels

Custom labels can be added to metrics to create more meaningful monitoring metrics. For example, a common ownedBy label across all the resources in a cluster, businessUnit or department name from the labels of objects, etc.

For example, if the GitRepository objects are labelled with department:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: foo
 namespace: bar
 labels:
 department: baz
spec:
 interval: 1h
 ref:
 branch: main
 url: https://github.com/fluxcd/flux2-monitoring-example

The KSM customResourceState value can be configured to extract the department name, owned by team name and Git branch name, and include them as labels in the gotk_resource_info metric of GitRepositories.

...
customResourceState:
 config:
 spec:
 resources:
 - groupVersionKind:
 group: source.toolkit.fluxcd.io
 version: "v1"
 kind: GitRepository
 metricNamePrefix: gotk
 commonLabels:
 ownedBy: teamA
 labelsFromPath:
 department: [metadata, labels, department]
 metrics:
 - name: "resource_info"
 help: "The current state of a GitOps Toolkit resource."
 each:
 type: Info
 info:
 labelsFromPath:
 name: [metadata, name]
 labelsFromPath:
 exported_namespace: [metadata, namespace]
 suspended: [spec, suspend]
 ready: [status, conditions, "[type=Ready]", status]
 branch: [spec, ref, branch]
...

The above configuration will result in the following metric

gotk_resource_info{branch="main",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1",department="baz",exported_namespace="bar",name="foo",ownedBy="teamA",ready="True"} 1

It contains the ownedBy="teamA", department="baz" and branch="main" labels. Similarly, more custom labels can be added depending on the need.

Refer to the kube-state-metrics custom-resource state configuration docs to learn more about customizing the metrics.

Flux: Flux logs

Mon, 01 Jan 0001 00:00:00 +0000

The Flux controllers follow the Kubernetes structured logging conventions. These logs can be collected and analyzed to monitor the operations of the controllers.

The fluxcd/flux2-monitoring-example repository provides a ready-made example setup to get started with monitoring Flux, which includes Loki-stack to collect logs from all the Flux controllers and explore them using Grafana. It is recommended to set up the monitoring example before continuing with this document to follow along. Before getting into Loki and Grafana setup, the following sections will describe the Flux logs and how to interpret them.

Controller logs

The default installation of Flux controllers write logs to stderr in JSON format at the info log level. This can be configured using the --log-encoding and --log-level flags in the controllers. Refer to the flux-system kustomization for an example of how to patch the Flux controllers with flags. The following example patch snippet can be appended to the existing set of patches to add a log level flag and change the log level of the controller to debug.

- op: add
 path: /spec/template/spec/containers/0/args/-
 value: --log-level="debug"

Note: The patch configuration in the example only applies to a few targeted controllers. Update the patch target to apply this to other controllers too.

Structured logging

The Flux controllers support structured logging with the following common labels:

level can be debug, info or error
ts timestamp in the ISO 8601 format
msg info or error description
error error details (present when level is error)
controllerGroup the Flux CR group
controllerKind the Flux CR kind
name The Flux CR name
namespace The Flux CR namespace
reconcileID the UID of the Flux reconcile operation

Sample of an info log produced by kustomize-controller:

{
 "level": "info",
 "ts": "2023-08-16T09:36:41.286Z",
 "controllerGroup": "kustomize.toolkit.fluxcd.io",
 "controllerKind": "Kustomization",
 "name": "redis",
 "namespace": "apps",
 "msg": "server-side apply completed",
 "revision": "main@sha1:30081ad7170fb8168536768fe399493dd43160d7",
 "output": {
 "ConfigMap/apps/redis": "created",
 "Deployment/apps/redis": "configured",
 "HorizontalPodAutoscaler/apps/redis": "deleted",
 "Service/apps/redis": "unchanged",
 "Secret/apps/redis": "skipped"
 }
}

Sample of an error log produced by kustomize-controller:

{
 "level": "error",
 "ts": "2023-08-16T09:36:41.286Z",
 "controllerGroup": "kustomize.toolkit.fluxcd.io",
 "controllerKind": "Kustomization",
 "name": "redis",
 "namespace": "apps",
 "msg": "Reconciliation failed after 2s, next try in 5m0s",
 "revision": "main@sha1:f68c334e0f5fae791d1e47dbcabed256f4f89e68",
 "error": "Service/apps/redis dry-run failed, reason: Invalid, error: Service redis is invalid: spec.type: Unsupported value: Ingress"
}

The log labels shown above can be used to query for specific types of logs. For example, error logs can be queried using the error label, the output of successful reconciliation of Kustomization can be queried using the output label, the logs about a specific controller can be queried using the controllerKind label.

Querying logs associated with resources

For querying logs associated with particular resources, the flux logs CLI command can be used. It connects to the cluster, fetches the relevant Flux logs, and filters them based on the query request. For example, to list the logs associated with Kustomization monitoring-configs:

$ flux logs --kind=Kustomization --name=monitoring-configs --namespace=flux-system --since=1m
...
2023-08-22T18:35:45.292Z info Kustomization/monitoring-configs.flux-system - All dependencies are ready, proceeding with reconciliation
2023-08-22T18:35:45.348Z info Kustomization/monitoring-configs.flux-system - server-side apply completed
2023-08-22T18:35:45.380Z info Kustomization/monitoring-configs.flux-system - Reconciliation finished in 88.208385ms, next run in 1h0m0s

Refer to the flux logs CLI docs to learn more about it.

Log aggregation with Grafana Loki

In the monitoring example repository, the monitoring configurations can be found in the monitoring/ directory. monitoring/controllers/ directory contains the configurations for deploying kube-prometheus-stack and loki-stack. We’ll discuss loki-stack below. For Flux metrics collection using Prometheus, refer to the Flux Prometheus metrics docs.

The configuration in the monitoring/controllers/loki-stack directory creates a HelmRepository for the Grafana helm-charts and a HelmRelease to deploy the loki-stack chart in the monitoring namespace. Please see the values used for the chart and modify them accordingly.

Once deployed, Loki and Promtail Pods get created, and Loki is added as a data source in Grafana. Promtail aggregates the logs from all the Pods in every node and sends them to Loki. Grafana can be used to query the logs from Loki and analyze them. Refer to the LogQL docs to see examples of queries and learn more about querying logs.

Grafana dashboard

The example monitoring setup provides a Grafana dashboard in monitoring/configs/dashboards/logs.json that queries and shows logs from all the Flux controllers.

Control plane logs:

This can be used to browse logs from all the Flux controllers in a centralized manner.

Flux: Flux events

Mon, 01 Jan 0001 00:00:00 +0000

The Flux controllers emit Kubernetes events during the reconciliation operation to provide information about the object being reconciled. Unlike logs, events are always associated with an object, which is a Flux resource in this case. Events are supplemental data that can be used along with logs to provide a complete picture of controllers’ operations. Some of the events emitted by Flux controllers are also used to send notifications. See the Alerts docs to learn more about the Flux Alerts based on events from controllers. In the following sections, we will go through the Flux events and how to interpret them.

Kubernetes events

The Flux controller events about a resource contain the following fields:

type can be Normal or Warning
firstTimestamp timestamp in the ISO 8601 format
lastTimestamp timestamp in the ISO 8601 format
message info or warning description
reason short machine understandable string
involvedObject the API version, kind, name and namespace of the Flux object
metadata.annotations the Flux specific metadata e.g. source revision
source.component the Flux controller name where the event originated from.

Examples

Example of a Normal event produced by kustomize-controller:

{
 "kind": "Event",
 "apiVersion": "v1",
 "metadata": {
 "name": "flux-system.177bd633e296a292",
 "namespace": "flux-system",
 "annotations": {
 "kustomize.toolkit.fluxcd.io/revision": "main@sha1:802723078affd3eb2a3898630261ab3ca5d6dd40"
 }
 },
 "involvedObject": {
 "kind": "Kustomization",
 "namespace": "flux-system",
 "name": "flux-system",
 "apiVersion": "kustomize.toolkit.fluxcd.io/v1",
 },
 "reason": "ReconciliationSucceeded",
 "message": "Reconciliation finished in 436.493292ms, next run in 10m0s",
 "source": {
 "component": "kustomize-controller"
 },
 "firstTimestamp": "2023-08-16T10:26:43Z",
 "lastTimestamp": "2023-08-16T10:26:43Z",
 "type": "Normal",
}

In the above example:

The event is about a Kustomization named flux-system in the flux-system namespace, indicated by the involvedObject field.
The event originates from kustomize-controller, indicated by the source.component field.
The event is a Normal type event about a successful reconciliation, indicated by the reason and message fields.
The metadata.annotations field kustomize.toolkit.fluxcd.io/revision contains information about the source revision that was successfully applied as a result of successful reconciliation of the Kustomization.

Example of a Warning event produced by source-controller:

{
 "apiVersion": "v1",
 "count": 4,
 "eventTime": null,
 "firstTimestamp": "2023-08-22T20:24:06Z",
 "involvedObject": {
 "apiVersion": "source.toolkit.fluxcd.io/v1",
 "kind": "GitRepository",
 "name": "podinfo",
 "namespace": "default",
 "resourceVersion": "1284973",
 "uid": "2c2ed1da-556f-4793-863d-7d96e8bab3f5"
 },
 "kind": "Event",
 "lastTimestamp": "2023-08-22T20:24:18Z",
 "message": "failed to checkout and determine revision: unable to clone 'https://github.com/stefanprodan/podinfo': couldn't find remote ref \"refs/tags/v1.8.9\"",
 "metadata": {
 "creationTimestamp": "2023-08-22T20:24:06Z",
 "name": "podinfo.177dce48bc7db3a4",
 "namespace": "default",
 "resourceVersion": "1285016",
 "uid": "3c8f568a-c99b-4279-8093-6ef08fae325b"
 },
 "reason": "GitOperationFailed",
 "reportingComponent": "",
 "reportingInstance": "",
 "source": {
 "component": "source-controller"
 },
 "type": "Warning"
}

In the above example:

The event is about a GitRepository named podinfo in the default namespace, indicated by the involvedObject field.
The event originates from source-controller, indicated by the source.component field.
The event is a Warning type event about a failed Git operation, indicated by the reason and message fields.

Events inspection with kubectl

The events associated with a Flux resource can be queried using kubectl events command:

$ kubectl events -n flux-system --for kustomization/flux-system
LAST SEEN TYPE REASON OBJECT MESSAGE
58m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 448.00332ms, next run in 10m0s
48m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 486.826649ms, next run in 10m0s
38m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 502.282127ms, next run in 10m0s
28m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 543.745587ms, next run in 10m0s
18m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 465.177441ms, next run in 10m0s
8m27s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 494.543068ms, next run in 10m0s

This shows all the events associated with the queried resource in an hour.

Events inspection with flux CLI

The events associated with a Flux resource can be queried using the flux events CLI command:

$ flux events --for Kustomization/flux-system
LAST SEEN TYPE REASON OBJECT MESSAGE
52m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 506.467ms, next run in 10m0s
42m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 531.072726ms, next run in 10m0
32m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 506.673992ms, next run in 10m0
22m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 512.255817ms, next run in 10m0
12m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 507.521248ms, next run in 10m0
2m31s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 448.00332ms, next run in 10m0s

This can also be used to watch all the events issues by the Flux controllers across all the namespaces:

$ flux events --all-namespaces --watch
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
flux-system 34m (x3 over 154m) Normal GitOperationSucceeded GitRepository/flux-system no changes since last reconcilation: observed revision 'main@sha1:4d768edba5d409feb60870dd3b0ac0d307299898'
flux-system 54m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 486.814878ms, next run in 10m0s
flux-system 44m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 486.203813ms, next run in 10m0s
flux-system 34m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 512.160373ms, next run in 10m0s
flux-system 24m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 543.806383ms, next run in 10m0s
flux-system 14m Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 524.293527ms, next run in 10m0s
flux-system 4m5s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 522.671955ms, next run in 10m0s
flux-system 47s Normal ReconciliationSucceeded Kustomization/flux-system Reconciliation finished in 523.892245ms, next run in 10m0s
flux-system 34m Normal ReconciliationSucceeded Kustomization/monitoring-configs Reconciliation finished in 104.609707ms, next run in 1h0m0s
flux-system 42s Normal ReconciliationSucceeded Kustomization/monitoring-configs Reconciliation finished in 90.70521ms, next run in 1h0m0s
flux-system 34m Normal ReconciliationSucceeded Kustomization/monitoring-controllers Reconciliation finished in 118.651968ms, next run in 1h0m0s
flux-system 39s Normal ReconciliationSucceeded Kustomization/monitoring-controllers Reconciliation finished in 132.34839ms, next run in 1h0m0s
monitoring 34m (x3 over 154m) Normal ArtifactUpToDate HelmChart/monitoring-kube-prometheus-stack artifact up-to-date with remote revision: '48.3.3'
monitoring 34m (x3 over 154m) Normal ArtifactUpToDate HelmChart/monitoring-loki-stack artifact up-to-date with remote revision: '2.9.11'

Refer to the flux events CLI docs to learn more about it.