Flux – Image reflector and automation controllers

Flux: Controller Options

Mon, 01 Jan 0001 00:00:00 +0000

To customise the controller options at install time, please see the bootstrap customization guide.

Image automation flags

Name	Type	Description
`--concurrent`	int	The number of concurrent kustomize reconciles. (default 4)
`--default-service-account`	string	Default service account to use for workload identity when not specified in resources.
`--enable-leader-election`	boolean	Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
`--events-addr`	string	The address of the events receiver.
`--health-addr`	string	The address the health endpoint binds to. (default “:9440”)
`--leader-election-lease-duration`	duration	Interval at which non-leader candidates will wait to force acquire leadership (duration string). (default 35s)
`--leader-election-release-on-cancel`	boolean	Defines if the leader should step down voluntarily on controller manager shutdown. (default true)
`--leader-election-renew-deadline`	duration	Duration that the leading controller manager will retry refreshing leadership before giving up (duration string). (default 30s)
`--leader-election-retry-period`	duration	Duration the LeaderElector clients should wait between tries of actions (duration string). (default 5s)
`--log-encoding`	string	Log encoding format. Can be ‘json’ or ‘console’. (default “json”)
`--log-level`	string	Log verbosity level. Can be one of ’trace’, ‘debug’, ‘info’, ’error’. (default “info”)
`--max-retry-delay`	duration	The maximum amount of time for which an object being reconciled will have to wait before a retry. (default 15m0s)
`--metrics-addr`	string	The address the metric endpoint binds to. (default “:8080”)
`--min-retry-delay`	duration	The minimum amount of time for which an object being reconciled will have to wait before a retry. (default 750ms)
`--no-cross-namespace-refs`	boolean	When set to true, references between custom resources are allowed only if the reference and the referee are in the same namespace.
`--ssh-hostkey-algos`	strings	The list of hostkey algorithms to use for ssh connections, arranged from most preferred to the least.
`--ssh-kex-algos`	strings	The list of key exchange algorithms to use for ssh connections, arranged from most preferred to the least.
`--token-cache-max-size`	int	The maximum amount of entries in the LRU cache used for tokens. (default 100, enabled)
`--token-cache-max-duration`	duration	The maximum duration for which a token would be considered unexpired. This is capped at 1h. (default 1h)
`--watch-all-namespaces`	boolean	Watch for custom resources in all namespaces, if set to false it will only watch the runtime namespace. (default true)
`--watch-label-selector`	string	Watch for resources with matching labels e.g. ‘sharding.fluxcd.io/key=shard1’.
`--feature-gates`	mapStringBool	A comma separated list of key=value pairs defining the state of experimental features.

Feature Gates

Name	Default Value	Description
`CacheSecretsAndConfigMaps`	`false`	Configures the caching of Secrets and ConfigMaps by the controller-runtime client. When enabled, it will cache both object types, resulting in increased memory usage and cluster-wide RBAC permissions (list and watch).
`GitAllBranchReferences`	`true`	Enables the download of all branch head references when push branches are configured.
`GitForcePushBranch`	`true`	Enables the use of “force push” when pushing changes to a separate branch. This fixes issues with stale push branches.
`GitSparseCheckout`	`false`	Enables the use of Git sparse checkout to only fetch the path defined in `.spec.update.path` from the repository.
`ObjectLevelWorkloadIdentity`	`false`	Enables the use of object-level workload identity for the controller.

Image reflector flags

Name	Type	Description
`--concurrent`	int	The number of concurrent kustomize reconciles. (default 4)
`--default-service-account`	string	Default service account to use for workload identity when not specified in resources.
`--enable-leader-election`	boolean	Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
`--events-addr`	string	The address of the events receiver.
`--gc-interval`	int	The interval in number of minutes at which the garbage collector will run for the tags database. Zero disables GC. (default 10)
`--health-addr`	string	The address the health endpoint binds to. (default “:9440”)
`--leader-election-lease-duration`	duration	Interval at which non-leader candidates will wait to force acquire leadership (duration string). (default 35s)
`--leader-election-release-on-cancel`	boolean	Defines if the leader should step down voluntarily on controller manager shutdown. (default true)
`--leader-election-renew-deadline`	duration	Duration that the leading controller manager will retry refreshing leadership before giving up (duration string). (default 30s)
`--leader-election-retry-period`	duration	Duration the LeaderElector clients should wait between tries of actions (duration string). (default 5s)
`--log-encoding`	string	Log encoding format. Can be ‘json’ or ‘console’. (default “json”)
`--log-level`	string	Log verbosity level. Can be one of ’trace’, ‘debug’, ‘info’, ’error’. (default “info”)
`--metrics-addr`	string	The address the metric endpoint binds to. (default “:8080”)
`--no-cross-namespace-refs`	boolean	When set to true, references between custom resources are allowed only if the reference and the referee are in the same namespace.
`--storage-path`	string	Where to store the persistent database of image metadata. (default “/data”)
`--storage-value-log-file-size`	int	Set the database’s memory mapped value log file size in bytes. Effective memory usage is about two times this size. (default 268435456)
`--token-cache-max-size`	int	The maximum amount of entries in the LRU cache used for tokens. (default 100, enabled)
`--token-cache-max-duration`	duration	The maximum duration for which a token would be considered unexpired. This is capped at 1h. (default 1h)
`--watch-all-namespaces`	boolean	Watch for custom resources in all namespaces, if set to false it will only watch the runtime namespace. (default true)
`--watch-label-selector`	string	Watch for resources with matching labels e.g. ‘sharding.fluxcd.io/key=shard1’.
`--feature-gates`	mapStringBool	A comma separated list of key=value pairs defining the state of experimental features.

Feature Gates

Name	Default Value	Description
`CacheSecretsAndConfigMaps`	`false`	Configures the caching of Secrets and ConfigMaps by the controller-runtime client. When enabled, it will cache both object types, resulting in increased memory usage and cluster-wide RBAC permissions (list and watch).
`ObjectLevelWorkloadIdentity`	`false`	Enables the use of object-level workload identity for the controller.

Flux: Image Policies

Mon, 01 Jan 0001 00:00:00 +0000

The ImagePolicies API defines rules for selecting a “latest” image from ImageRepositories.

Example

The following is an example of an ImagePolicy. It queries the referred ImageRepository for the image name of the repository, reads all the tags in the repository and selects the latest tag based on the defined policy rules.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
 namespace: default
spec:
 imageRepositoryRef:
 name: podinfo
 digestReflectionPolicy: IfNotPresent
 policy:
 semver:
 range: 5.1.x

In the above example:

An ImagePolicy named podinfo is created, indicated by the .metadata.name field.
The image-reflector-controller applies the latest tag selection policy every time there’s an update in the referred ImageRepository, indicated by the .spec.imageRepositoryRef.name field.
It fetches the canonical image name of the referred ImageRepository and reads the scanned tags from the internal database for the image name. The read tags are then used to select the latest tag based on the policy defined in .spec.policy.
The latest image’s name is derived from the ImageRepository image and reported together with the selected tag and digest in the .status.latestRef object.

This example can be run by saving the manifest into imagepolicy.yaml.

Apply the resource on the cluster:

kubectl apply -f imagepolicy.yaml

Run kubectl get imagepolicy to see the ImagePolicy:

NAME IMAGE TAG READY STATUS AGE
podinfo ghcr.io/stefanprodan/podinfo 5.1.4 True Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4 5m

Run kubectl describe imagepolicy podinfo to see the Latest Ref and Conditions in the ImagePolicy’s Status:

Status:
 Conditions:
 Last Transition Time: 2022-09-20T07:09:56Z
 Message: Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4
 Observed Generation: 1
 Reason: Succeeded
 Status: True
 Type: Ready
 Latest Ref:
 Digest: sha256:2d9a00b3981628a533ff43352193b1838b0a4bf6b0033444286f563205e51a2c
 Image: ghcr.io/stefanprodan/podinfo
 Tag: 5.1.4
 Observed Generation: 1
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Succeeded 7s (x3 over 8s) image-reflector-controller Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4

Writing an ImagePolicy spec

As with all other Kubernetes config, an ImagePolicy needs apiVersion, kind, and metadata fields. The name of an ImagePolicy object must be a valid DNS subdomain name.

An ImagePolicy also needs a .spec section.

Image Repository Reference

.spec.imageRepositoryRef is a required field that specifies the ImageRepository for which the latest image has to be selected. The value must be a namespaced object reference. For ImageRepository in the same namespace as the ImagePolicy, no namespace needs to be provided. For ImageRepository in a different namespace than the namespace of the ImagePolicy, namespace name has to be provided. For example:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
 namespace: default
spec:
 imageRepositoryRef:
 name: podinfo
 namespace: flux-system
...

The ImageRepository access is determied by its ACL for cross-namespace reference. For more details on how to allow cross-namespace references see the ImageRepository docs.

Policy

.spec.policy is a required field that specifies how to choose a latest image given the image metadata. There are three image policy choices:

SemVer
Alphabetical
Numerical

SemVer

SemVer policy interprets all the tags as semver versions and chooses the highest version available that fits the given semver constraints. The constraint is set in the .spec.policy.semver.range field.

Example of a SemVer image policy choice:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
spec:
 imageRepositoryRef:
 name: podinfo
 policy:
 semver:
 range: '>=1.0.0'

This will select the latest stable version tag.

Alphabetical

Alphabetical policy chooses the last tag when all the tags are sorted alphabetically (in either ascending or descending order). The sort order is set in the .spec.policy.alphabetical.order field. The value could be asc for ascending order or desc for descending order. The default value is asc.

Example of an Alphabetical policy choice:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
spec:
 imageRepositoryRef:
 name: podinfo
 policy:
 alphabetical:
 order: asc

This will select the last tag when all the tags are sorted alphabetically in ascending order.

Numerical

Numerical policy chooses the last tag when all the tags are sorted numerically (in either ascending or descending order). The sort order is set in the .spec.policy.numerical.order field. The value could be asc for ascending order or desc for descending order. The default value is asc.

Example of a Numerical policy choice:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
spec:
 imageRepositoryRef:
 name: podinfo
 policy:
 numerical:
 order: asc

This will select the last tag when all the tags are sorted numerically in ascending order.

Filter Tags

.spec.filterTags is an optional field to specify a filter on the image tags before they are considered by the policy rule.

The filter pattern is a regular expression, set in the .spec.filterTags.pattern field. Only tags that match the pattern are considered by the policy rule.

The .spec.filterTags.extract is an optional field used to extract a value from the matching tags which is supplied to the policy rule instead of the original tags. If unspecified, the tags that match the pattern will be used as they are.

Example of selecting the latest release candidate (semver):

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
spec:
 imageRepositoryRef:
 name: podinfo
 filterTags:
 pattern: '.*-rc.*'
 policy:
 semver:
 range: '^1.x-0'

Example of selecting the latest release tagged as RELEASE.<RFC3339-TIMESTAMP> (alphabetical):

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: minio
spec:
 imageRepositoryRef:
 name: minio
 filterTags:
 pattern: '^RELEASE\.(?P<timestamp>.*)Z$'
 extract: '$timestamp'
 policy:
 alphabetical:
 order: asc

In the above example, the timestamp value from the tag pattern is extracted and used in the policy rule to determine the latest tag.

Digest Reflection

.spec.digestReflectionPolicy is a field that governs the reflection of the selected image’s digest in the ImagePolicy’s .status.latestRef.digest field. The field has three possible values:

Never: If the field is set to Never (the default) the digest will not be reflected at all.
Always: This value leads to the digest of the latest tag to always be reflected in .status.latestRef.digest. An existing, potentially different digest will be overwritten with the most recent value retrieved from the image registry even if the tag didn’t change. This may be useful to track mutable tags like latest.
IfNotPresent: This value will only store the digest of the latest tag once and never overwrite an existing value unless the tag has changed as well. This is the safest option to track immutable tags.

Interval

.spec.interval specifies the interval at which the ImagePolicy must refresh the digest of the latest tag. The value must be in a Go recognized duration string format, e.g. 10m0s to reconcile the object every 10 minutes. This field must and can only be specified when .spec.digestReflectionPolicy is set to Always.

Working with ImagePolicy

Triggering a reconcile

ImagePolicy is reconciled automatically when the associated ImageRepository is updated. Whenever ImageRepository gets updated, ImagePolicy will be triggered and have the policy result based on the latest values of ImageRepository. To manually tell the image-reflector-controller to reconcile an ImagePolicy, the associated ImageRepository can be annotated with reconcile.fluxcd.io/requestedAt: <arbitrary value>. See triggering a reconcile for more details about reconciling ImageRepository.

Waiting for `Ready`

When a change is applied, it is possible to wait for the ImagePolicy to reach a ready state using kubectl:

kubectl wait imagepolicy/<policy-name> --for=condition=ready --timeout=1m

Debugging an ImagePolicy

There are several ways to gather information about an ImagePolicy for debugging purposes.

Describe the ImagePolicy

Describing an ImagePolicy using kubectl describe imagepolicy <policy-name> displays the latest recorded information for the resource in the Status and Events sections:

...
Status:
 Conditions:
 Last Transition Time: 2022-10-06T12:07:35Z
 Message: accessing ImageRepository
 Observed Generation: 1
 Reason: AccessingRepository
 Status: True
 Type: Reconciling
 Last Transition Time: 2022-10-06T12:07:35Z
 Message: failed to get the referred ImageRepository: referenced ImageRepository does not exist: ImageRepository.image.toolkit.fluxcd.io "podinfo" not found
 Observed Generation: 1
 Reason: DependencyNotReady
 Status: False
 Type: Ready
 Observed Generation: 1
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Warning DependencyNotReady 2s (x4 over 5s) image-reflector-controller failed to get the referred ImageRepository: referenced ImageRepository does not exist: ImageRepository.image.toolkit.fluxcd.io "podinfo" not found

Trace emitted Events

To view events for specific ImagePolicy(s), kubectl events can be used in combination with --for to list the Events for specific objects. For example, running

kubectl events --for ImagePolicy/<policy-name>

lists

LAST SEEN TYPE REASON OBJECT MESSAGE
4m44s Normal Succeeded imagepolicy/<policy-name> Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4
95s Warning DependencyNotReady imagepolicy/<policy-name> failed to get the referred ImageRepository: referenced ImageRepository does not exist: ImageRepository.image.toolkit.fluxcd.io "podinfo" not found

Besides being reported in Events, the reconciliation errors are also logged by the controller. The Flux CLI offer commands for filtering the logs for a specific ImagePolicy, e.g. flux logs --level=error --kind=ImagePolicy --name=<policy-name>.

ImagePolicy Status

Latest Ref

The ImagePolicy reports the latest selected image from the ImageRepository tags in .status.latestRef for the resource. The field .status.latestRef.digest is dependent on the chosen digest reflection policy and is only set for the Always or IfNotPresent policies.

Example:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: <policy-name>
status:
 latestRef:
 image: ghcr.io/stefanprodan/podinfo
 tag: 5.1.4
 digest: sha256:2d9a00b3981628a533ff43352193b1838b0a4bf6b0033444286f563205e51a2c
[...]

Observed Previous Ref

The ImagePolicy reports the previously observed latest image in .status.observedPreviousRef for the resource. This is used by the ImagePolicy to determine an upgrade path of an ImagePolicy update. This field is reset when the ImagePolicy fails due to some reason to be able to distinguish between a failure recovery and a genuine latest image upgrade.

Example:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: <policy-name>
status:
 latestRef:
 image: ghcr.io/stefanprodan/podinfo
 tag: 6.2.1
 observedPreviousRef:
 image: ghcr.io/stefanprodan/podinfo
 tag: 5.1.4

Conditions

An ImagePolicy enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be reconciling while reading the tags from ImageRepository scan results, it can be ready, or it can fail during reconciliation.

The ImagePolicy API is compatible with the kstatus specification, and reports Reconciling and Stalled conditions where applicable to provide better (timeout) support to solutions polling the ImagePolicy to become Ready.

Reconciling ImagePolicy

The image-reflector-controller marks an ImagePolicy as reconciling when one of the following is true:

The generation of the ImagePolicy is newer than the Observed Generation.
The ImagePolicy is accessing the provided ImageRepository reference.
The ImagePolicy is being applied to the tags read from an ImageRepository.

When the ImagePolicy is “reconciling”, the Ready Condition status becomes False, and the controller adds a Condition with the following attributes to the ImagePolicy’s .status.conditions:

type: Reconciling
status: "True"
reason: NewGeneration | reason:AccessingRepository | reason: ApplyingPolicy

It has a “negative polarity”, and is only present on the ImagePolicy while its status value is "True".

Ready ImagePolicy

The image-reflector-controller marks an ImagePolicy as ready when it has the following characteristics:

The ImagePolicy reports a Latest Image
The referenced ImageRepository is accessible and the internal tags database contains the tags that ImagePolicy needs to apply the policy on.

When the ImagePolicy is “ready”, the controller sets a Condition with the following attributes in the ImagePolicy’s .status.conditions.

type: Ready
status: "True"
reason: Succeeded

This Ready Condition will retain a status value of "True" until the ImagePolicy is marked as reconciling, or e.g. a transient error occurs due to a temporary network issue.

Failed ImagePolicy

The image-reflector-controller may get stuck trying to apply a policy without completing. This can occur due to some of the following factors:

The referenced ImageRepository is temporarily unavailable.
The referenced ImageRepository does not exist.
The referenced ImageRepository is not accessible in a different namespace.
The ImagePolicy spec contains a generic misconfiguration.
The ImagePolicy could not select the latest tag based on the given rules and the available tags.
A database related failure when reading or writing the scanned tags.

When this happens, the controller sets the Ready condition status to False wit the following reason:

reason: Failure | reason: AccessDenied | reason: DependencyNotReady

While the ImagePolicy is in failing state, the controller will continue to attempt to get the referenced ImageRepository for the resource and apply the policy rules with an exponential backoff, until it succeeds and the ImagePolicy is marked as ready.

Note that an ImagePolicy can be reconcilcing while failing at the same time, for example due to a newly introduced configuration issue in the ImagePolicy spec.

Observed Generation

The image-reflector-controller reports an observed generation in the ImagePolicy’s .status.observedGeneration. The observed generation is the latest .metadata.generation which resulted in either a ready state, or stalled due to error it can not recover from without human intervention.

Flux: Image Repositories

Mon, 01 Jan 0001 00:00:00 +0000

The ImageRepository API defines a repository to scan and store a specific set of tags in a database.

Example

The following is an example of an ImageRepository. It scans the specified image repository and stores the scanned tags in an internal database.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: podinfo
 namespace: default
spec:
 image: stefanprodan/podinfo
 interval: 1h
 provider: generic

In the above example:

An ImageRepository named podinfo is created, indicated by the .metadata.name field.
The image-reflector-controller scans the image repository for tags every hour, indicated by the .spec.interval field.
The registry authentication is done using a generic provider, indicated by the .spec.provider field and referenced using .spec.secretRef. No authentication is attempted when secret reference is not provided for generic provider. See Provider for more details related to registry authentication.
The canonical form of the image set in .spec.image is used to scan the repository. The resolved canonical form of the image is reported in the .status.canonicalImageName field.
The result of the scan is reported in the .status.lastScanResult field.

This example can be run by saving the manifest into imagerepository.yaml.

Apply the resource on the cluster:

kubectl apply -f imagerepository.yaml

Run kubectl get imagerepository to see the ImageRepository:

NAME LAST SCAN TAGS
podinfo 2022-09-15T22:34:05Z 211

Run kubectl describe imagerepository podinfo to see the Last Scan Result and Conditions in the ImageRepository’s Status:


...
Status:
 Canonical Image Name: index.docker.io/stefanprodan/podinfo
 Conditions:
 Last Transition Time: 2022-09-15T22:38:42Z
 Message: successful scan, found 211 tags
 Observed Generation: 1
 Reason: Succeeded
 Status: True
 Type: Ready
 Last Scan Result:
 Latest Tags:
 latest
 6.2.0
 6.1.8
 6.1.7
 6.1.6
 6.1.5
 6.1.4
 6.1.3
 6.1.2
 6.1.1
 Scan Time: 2022-09-15T22:38:42Z
 Tag Count: 211
 Observed Exclusion List:
 ^.*\.sig$
 Observed Generation: 1
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Succeeded 17s image-reflector-controller successful scan, found 211 tags

Writing an ImageRepository spec

As with all other Kubernetes config, an ImageRepository needs apiVersion, kind, and metadata fields. The name of an ImageRepository object must be a valid DNS subdomain name.

An ImageRepository also needs a .spec section.

Image

.spec.image is a required field that specifies the address of an image repository without any scheme prefix, e.g. fluxcd/image-reflector-controller. This image is converted to its canonical form by the controller before scanning. The canonical form of the image is reflected in .status.canonicalImageName.

Interval

.spec.interval is a required field that specifies the interval at which the Image repository must be scanned.

After successfully reconciling the object, the image-reflector-controller requeues it for inspection after the specified interval. The value must be in a Go recognized duration string format, e.g. 10m0s to reconcile the object every 10 minutes.

If the .metadata.generation of a resource changes (due to e.g. a change to the spec), this is handled instantly outside the interval window.

Timeout

.spec.timeout is an optional field to specify a timeout for various operations during the reconciliation like fetching the referred secrets, scanning the repository, etc. The value must be in a Go recognized duration string format, e.g. 1m30s for a timeout of one minute and thirty seconds. The default value is the value of .spec.interval.

Secret reference

.spec.secretRef.name is an optional field to specify a name reference to a Secret in the same namespace as the ImageRepository, containing authentication credentials for the Image repository. The secret is expected to be in the same format as the docker config secrets, usually created by kubectl create secret docker-registry.

Example of using secret reference in an ImageRepository:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: podinfo
 namespace: default
spec:
 image: stefanprodan/podinfo
 interval: 1h
 secretRef:
 name: regcred
---
apiVersion: v1
kind: Secret
metadata:
 name: regcred
 namespace: default
type: kubernetes.io/dockerconfigjson
data:
 .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2luZGV4LmRvY2tlci5pby92MS8iOnsidXNlcm5hbWUiOiJmb28iLCJwYXNzd29yZCI6ImJhciIsImF1dGgiOiJabTl2T21KaGNnPT0ifX19

For a publicly accessible image repository, there’s no need to provide a secret reference.

ServiceAccount name

.spec.serviceAccountName is an optional field to specify a Service Account in the same namespace as ImageRepository with purpose depending on the value of the .spec.provider field:

When .spec.provider is set to generic, the controller will fetch the image pull secrets attached to the Service Account and use them for authentication.
When .spec.provider is set to aws, azure, or gcp, the Service Account will be used for Workload Identity authentication. In this case, the controller feature gate ObjectLevelWorkloadIdentity must be enabled, otherwise the controller will error out.

Note: that for a publicly accessible image repository, you don’t need to provide a secretRef nor serviceAccountName.

For a complete guide on how to set up authentication for cloud providers, see the integration docs.

Certificate secret reference

.spec.certSecretRef.name is an optional field to specify a secret containing TLS certificate data for secure communication. The secret must be of type Opaque or kubernetes.io/tls.

Supported configurations

Mutual TLS (mTLS): Client certificate authentication (provide tls.crt + tls.key, optionally with ca.crt)
CA-only: Server authentication (provide ca.crt only)

Mutual TLS Authentication

Mutual TLS authentication allows for secure client-server communication using client certificates stored in Kubernetes secrets. Both tls.crt and tls.key must be specified together for client certificate authentication. The ca.crt field is optional but required when connecting to servers with self-signed certificates.

All the files in the Secret are expected to be PEM-encoded. Assuming you have three files; client.key, client.crt and ca.crt for the client private key, client certificate and the CA certificate respectively, you can generate the required Secret using the flux create secret tls command:

flux create secret tls --tls-key-file=client.key --tls-crt-file=client.crt --ca-crt-file=ca.crt

Example: mTLS Configuration

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: example-mtls
 namespace: default
spec:
 interval: 5m0s
 image: example.com
 certSecretRef:
 name: example-mtls-certs
---
apiVersion: v1
kind: Secret
metadata:
 name: example-mtls-certs
 namespace: default
type: kubernetes.io/tls # or Opaque
stringData:
 tls.crt: |
 -----BEGIN CERTIFICATE-----
 <client certificate>
 -----END CERTIFICATE-----
 tls.key: |
 -----BEGIN PRIVATE KEY-----
 <client private key>
 -----END PRIVATE KEY-----
 ca.crt: |
 -----BEGIN CERTIFICATE-----
 <certificate authority certificate>
 -----END CERTIFICATE-----

CA Certificate Authentication

CA certificate authentication provides server authentication when connecting to container registries with self-signed or custom CA certificates. Only the ca.crt field is required for this configuration.

Example: CA Certificate Configuration

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: example-ca
 namespace: default
spec:
 interval: 5m0s
 image: example.com
 certSecretRef:
 name: example-ca-cert
---
apiVersion: v1
kind: Secret
metadata:
 name: example-ca-cert
 namespace: default
type: kubernetes.io/tls # or Opaque
stringData:
 ca.crt: |
 -----BEGIN CERTIFICATE-----
 <certificate authority certificate>
 -----END CERTIFICATE-----

Warning: Support for the caFile, certFile and keyFile keys have been deprecated. If you have any Secrets using these keys and specified in an ImageRepository, the controller will log a deprecation warning.

Proxy secret reference

.spec.proxySecretRef.name is an optional field used to specify the name of a Secret that contains the proxy settings for the object. These settings are used for all the remote operations related to the ImageRepository. The Secret may contain three keys:

address, to specify the address of the proxy server. This is a required key.
username, to specify the username to use if the proxy server is protected by basic authentication. This is an optional key.
password, to specify the password to use if the proxy server is protected by basic authentication. This is an optional key.

Example:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: example
 namespace: default
spec:
 interval: 5m0s
 image: example.com
 proxySecretRef:
 name: http-proxy
---
apiVersion: v1
kind: Secret
metadata:
 name: http-proxy
type: Opaque
stringData:
 address: http://proxy.com
 username: mandalorian
 password: grogu

Proxying can also be configured in the image-reflector-controller Deployment directly by using the standard environment variables such as HTTPS_PROXY, ALL_PROXY, etc.

.spec.proxySecretRef.name takes precedence over all environment variables.

Suspend

.spec.suspend is an optional field to suspend the reconciliation of an ImageRepository. When set to true, the controller will stop reconciling the ImageRepository, and changes to the resource or image repository will not result in new scan results. When the field is set to false or removed, it will resume.

Access from

.spec.accessFrom is an optional field to restrict cross-namespace access of ImageRepositories. To grant access to an ImageRepository for policies in other namespaces, the owner of the ImageRepository has to specify a list of label selectors that match the namespace labels of the ImagePolicy objects.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: app1
 namespace: apps
spec:
 interval: 1h
 image: docker.io/org/image
 secretRef:
 name: regcred
 accessFrom:
 namespaceSelectors:
 - matchLabels:
 kubernetes.io/metadata.name: flux-system

Note: The kubernetes.io/metadata.name label above is a readonly label added by Kubernetes >= 1.21 automatically on namespaces. For older version of Kubernetes, please set labels on the namespaces where the ImagePolicy exist.

The above definition, allows ImagePolicy in the flux-system namespace to reference the app1 ImageRepository e.g.:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: app1
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: app1
 namespace: apps
 policy:
 semver:
 range: 1.0.x

To grant access to all namespaces, an empty matchLabels can be set:

 accessFrom:
 namespaceSelectors:
 - matchLabels: {}

Exclusion list

.spec.exclusionList is an optional field to exclude certain tags in the image scan result. It’s a list of regular expression patterns with a default value of "^.*\\.sig$" if it’s not set. This default value is used to exclude all the tags ending with .sig, since these are Cosign generated objects and not container images which can be deployed on a Kubernetes cluster.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: app1
 namespace: apps
spec:
 interval: 1h
 image: docker.io/org/image
 exclusionList:
 - "^.*\\.sig$"
 - "1.0.2"
 - "1.1.1|1.0.0"

Insecure

.spec.insecure is an optional field to allow connecting to a non-TLS HTTP container registry.

Provider

.spec.provider is an optional field that allows specifying an OIDC provider used for authentication purposes.

Supported options are:

generic
aws
azure
gcp

The generic provider can be used for public repositories or when static credentials are used for authentication, either with spec.secretRef or spec.serviceAccountName. If you do not specify .spec.provider, it defaults to generic.

For a complete guide on how to set up authentication for cloud providers, see the integration docs.

AWS

The aws provider can be used to authenticate automatically using the EKS worker node IAM role or IAM Role for Service Accounts (IRSA), and by extension gain access to ECR.

Worker Node IAM

When the worker node IAM role has access to ECR, image-reflector-controller running on it will also have access to ECR. Please take a look at this documentation for creating worker node IAM roles.

IAM roles for service accounts(IRSA)

When using IRSA to enable access to ECR, add the following patch to your bootstrap repository, in the flux-system/kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: image-reflector-controller
 annotations:
 eks.amazonaws.com/role-arn: <role arn>
 target:
 kind: ServiceAccount
 name: image-reflector-controller

Note that you can attach the AWS managed policy arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly to the IAM role when using IRSA and you have to configure the image-reflector-controller to assume the IAM role. Please see documentation.

Note when you change the IAM role for the service account, you will need to restart the image-reflector-controller pod to use the new role. This is always true for any controller running on EKS.

kubectl rollout restart deployment -n flux-system image-reflector-controller

Azure

The azure provider can be used to authenticate automatically using Workload Identity or kubelet managed identity and by extension gain access to ACR.

Kubelet Identity

When the kubelet managed identity has access to ACR, image-reflector-controller running on it will also have access to ACR.

Workload Identity

When using workload identity to enable access to ACR, add the following patch to properly annotate the image-reflector-controller pods and service account in the flux-system/kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |-
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: image-reflector-controller
 namespace: flux-system
 annotations:
 azure.workload.identity/client-id: <AZURE_CLIENT_ID>
 labels:
 azure.workload.identity/use: "true"
 - patch: |-
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: image-reflector-controller
 namespace: flux-system
 labels:
 azure.workload.identity/use: "true"
 spec:
 template:
 metadata:
 labels:
 azure.workload.identity/use: "true"

To use workload identity on your cluster, you would have to install workload in your cluster, create an identity that has AcrPull role to ACR and establish azure federated identity between the identity and the image-reflector-controller service account. Please, take a look at the Azure documentation for Workload identity.

GCP

The gcp provider can be used to authenticate automatically using OAuth scopes or Workload Identity, and by extension gain access to GCR or Artifact Registry.

Access scopes

When the GKE nodes have the appropriate OAuth scope for accessing GCR and Artifact Registry, image-reflector-controller running on it will also have access to them.

Workload Identity

When using Workload Identity to enable access to GCR or Artifact Registry, add the following patch to your bootstrap repository, in the flux-system/kustomization.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: image-reflector-controller
 annotations:
 iam.gke.io/gcp-service-account: <identity-name>
 target:
 kind: ServiceAccount
 name: image-reflector-controller

The Artifact Registry service uses the permission artifactregistry.repositories.downloadArtifacts that is located under the Artifact Registry Reader role. If you are using Google Container Registry service, the needed permission is instead storage.objects.list which can be bound as part of the Container Registry Service Agent role. Take a look at this guide for more information about setting up GKE Workload Identity.

Authentication on other platforms

For other platforms that link service permissions to service accounts, secret can be created using tooling for that platform, rather than directly with kubectl create secret. There is advice specific to some platforms in the image automation guide.

Working with ImageRepositories

Triggering a reconcile

To manually tell the image-reflector-controller to reconcile an ImageRepository outside the specified interval window, an ImageRepository can be annotated with reconcile.fluxcd.io/requestedAt: <arbitrary value>. Annotating the resource queues the ImageRepository for reconciliation if the <arbitrary-value> differs from the last value the controller acted on, as reported in .status.lastHandledReconcileAt.

Using kubectl:

kubectl annotate --field-manager=flux-client-side-apply --overwrite imagerepository/<repository-name> reconcile.fluxcd.io/requestedAt="$(date +%s)"

Using flux:

flux reconcile image repository <repository-name>

Waiting for `Ready`

When a change is applied, it is possible to wait for the ImageRepository to reach a ready state using kubectl:

kubectl wait imagerepository/<repository-name> --for=condition=ready --timeout=1m

Suspending and resuming

When you find yourself in a situation where you temporarily want to pause the reconciliation of a ImageRepository, you can suspend it using the .spec.suspend field.

Suspend an ImageRepository

In your YAML declaration:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
 name: <repository-name>
spec:
 suspend: true

Using kubectl:

kubectl patch imagerepository <repository-name> --field-manager=flux-client-side-apply -p '{\"spec\": {\"suspend\" : true }}'

Using flux:

flux suspend image repository <repository-name>

Note: When an ImageRepository has scan results and is suspended, and this result later disappears from the database due to e.g. the image-reflector-controller Pod being evicted from a Node, this will not be reflected in the ImageRepository’s Status until it is resumed.

Resume an ImageRepository

In your YAML declaration, comment out (or remove) the .spec.suspend field:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
 name: <repository-name>
spec:
 # suspend: true

Note: Setting the field value to false has the same effect as removing it, but does not allow for “hot patching” using e.g. kubectl while practicing GitOps; as the manually applied patch would be overwritten by the declared state in Git.

Using kubectl:

kubectl patch imagerepository <repository-name> --field-manager=flux-client-side-apply -p '{\"spec\" : {\"suspend\" : false }}'

Using flux:

flux resume image repository <repository-name>

Debugging an ImageRepository

There are several ways to gather information about an ImageRepository for debugging purposes.

Describe the ImageRepository

Describing an ImageRepository using kubectl describe imagerepository <repository-name> displays the latest recorded information for the resource in the Status and Events sections:

...
Status:
 Conditions:
 Last Transition Time: 2022-09-19T05:47:40Z
 Message: could not parse reference: ghcr.io/stefanprodan/podinfo:foo:bar
 Observed Generation: 1
 Reason: ImageURLInvalid
 Status: True
 Type: Stalled
 Last Transition Time: 2022-09-19T05:47:40Z
 Message: could not parse reference: ghcr.io/stefanprodan/podinfo:foo:bar
 Observed Generation: 1
 Reason: ImageURLInvalid
 Status: False
 Type: Ready
 Observed Generation: 1
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Warning ImageURLInvalid 5s image-reflector-controller could not parse reference: ghcr.io/stefanprodan/podinfo:foo:bar

Trace emitted Events

To view events for specific ImageRepository(s), kubectl events can be used in combination with --for to list the Events for specific objects. For example, running

kubectl events --for ImageRepository/<repository-name>

lists

LAST SEEN TYPE REASON OBJECT MESSAGE
3m51s Normal Succeeded imagerepository/<repository-name> successful scan, found 34 tags
114s Warning ImageURLInvalid imagerepository/<repository-name> could not parse reference: ghcr.io/stefanprodan/podinfo:foo:bar

Besides being reported in Events, the reconciliation errors are also logged by the controller. The Flux CLI offer commands for filtering the logs for a specific ImageRepository, e.g. flux logs --level=error --kind=ImageRepository --name=<repository-name>.

ImageRepository Status

Last Scan Result

The ImageRepository reports the latest scanned tags from the image repository in .status.lastScanResult for the resource. The tags are stored in an internal database. .status.lastScanResult.scanTime shows the time of last scan. .status.lastScanResult.tagCount shows the number of tags in the result. This is calculated after applying any exclusion list rules.

Example:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: <repository-name>
status:
 lastScanResult:
 latestTags:
 - latest
 - 6.2.0
 - 6.1.8
 - 6.1.7
 - 6.1.6
 - 6.1.5
 - 6.1.4
 - 6.1.3
 - 6.1.2
 - 6.1.1
 scanTime: "2022-09-19T05:53:27Z"
 tagCount: 34

Canonical Image Name

The ImageRepository reports the canonical form of the image repository provided in the ImageRepository’s .spec.image in .status.canonicalImageName. Canonical name is the name of the image repository with all the implied bits made explicit; e.g., docker.io/library/alpine rather than alpine.

Observed Exclusion List

The ImageRepository reports an observed exclusion list in the ImageRepository’s .status.observedExclusionList. The observed exclusion list is the latest .spec.exclusionList which resulted in a ready state, or stalled due to error it can not recover from without human intervention.

Conditions

An ImageRepository enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be reconciling while scanning the image repository, it can be ready, or it can fail during reconciliation.

The ImageRepository API is compatible with the kstatus specification, and reports Reconciling and Stalled conditions where applicable to provide better (timeout) support to solutions polling the ImageRepository to become Ready.

Reconciling ImageRepository

The image-reflector-controller marks an ImageRepository as reconciling when one of the following is true:

The generation of the ImageRepository is newer than the Observed Generation.
The ImageRepository is being scanned because it’s scan time as per the specified spec.interval, or the ImageRepository has never been scanned before, or the reported tags in the last scanned results have disappeared from the database.

When the ImageRepository is “reconciling”, the Ready Condition status becomes False, and the controller adds a Condition with the following attributes to the ImageRepository’s .status.conditions:

type: Reconciling
status: "True"
reason: NewGeneration | reason: Scanning

It has a “negative polarity”, and is only present on the ImageRepository while its status value is "True".

Ready ImageRepository

The image-reflector-controller marks an ImageRepository as ready when it has the following characteristics:

The ImageRepository reports a Last Scan Result.
The reported tags exists in the controller’s internal database.
The controller was able to communicate with the remote image repository using the current spec.

When the ImageRepository is “ready”, the controller sets a Condition with the following attributes in the ImageRepository’s .status.conditions:

type: Ready
status: "True"
reason: Succeeded

This Ready Condition will retain a status value of "True" until the ImageRepository is marked as reconciling, or e.g. a transient error occurs due to a temporary network issue.

Failed ImageRepository

The image-reflector-controller may get stuck trying to scan an image repository without completing. This can occur due to some of the following factors:

The remote image repository is temporarily unavailable.
The image repository does not exist.
The Secret reference and Certificate secret reference contains a reference to a non-existing Secret.
The credentials and certificate in the referenced Secret are invalid.
The ImageRepository spec contains a generic misconfiguration.
A database related failure when reading or writing the scanned tags.

When this happens, the controller sets the Ready Condition status to False with the following reasons:

reason: ImageURLInvalid | reason: AuthenticationFailed | reason: Failure | reason: ReadOperationFailed

While the ImageRepository is in failing state, the controller will continue to attempt to scan the image repository for the resource with an exponential backoff, until it succeeds and the ImageRepository is marked as ready.

Note that an ImageRepository can be reconciling while failing at the same time, for example due to a newly introduced configuration issue in the ImageRepository spec.

Observed Generation

The image-reflector-controller reports an observed generation in the ImageRepository’s .status.observedGeneration. The observed generation is the latest .metadata.generation which resulted in either a ready state, or stalled due to error it can not recover from without human intervention.

Last Handled Reconcile At

The image-reflector-controller reports the last reconcile.fluxcd.io/requestedAt annotation value it acted on in the .status.lastHandledReconcileAt field.

For practical information about this field, see triggering a reconcile.

Flux: Image Reflector API Reference

Mon, 01 Jan 0001 00:00:00 +0000

Flux: Image Update Automation API Reference

Mon, 01 Jan 0001 00:00:00 +0000

Flux: Image Update Automations

Mon, 01 Jan 0001 00:00:00 +0000

The ImageUpdateAutomation API defines an automation process that will update a Git repository, based on ImagePolicy objects in the same namespace.

The updates are governed by marking fields to be updated in each YAML file. For each field marked, the automation process checks the image policy named, and updates the field value if there is a new image selected by the policy. The marker format is shown in the image automation guide.

Example

The following is an example of keeping the images in a Git repository up-to-date:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: podinfo
 namespace: default
spec:
 interval: 5m0s
 url: https://github.com/fluxcd/example
 ref:
 branch: main
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: podinfo
 namespace: default
spec:
 image: ghcr.io/stefanprodan/podinfo
 interval: 5h
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo-policy
 namespace: default
spec:
 imageRepositoryRef:
 name: podinfo
 policy:
 semver:
 range: 5.0.x
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: podinfo-update
 namespace: default
spec:
 interval: 30m
 sourceRef:
 kind: GitRepository
 name: podinfo
 git:
 commit:
 author:
 email: fluxcdbot@users.noreply.github.com
 name: fluxcdbot
 push:
 branch: main
 update:
 path: ./

In the above example:

A GitRepository named podinfo is created, indicated by the GitRepository.metadata.name field. The Git repository at https://github.com/fluxcd/example is assumed to contain YAML files with image policy markers, as described in image automation guide, to update them.
An ImageRepository named podinfo is created, indicated by the ImageRepository.metadata.name field. This scans all the tags for an image repository.
An ImagePolicy named podinfo-policy is created, indicated by the ImagePolicy.metadata.name field.
An ImageUpdateAutomation named podinfo-update is created, indicated by the ImageUpdateAutomation.metadata.name field.
The ImagePolicy refers to the podinfo ImageRepository to query for all the tags related to an image, indicated by ImagePolicy.spec.imageRepositoryRef. These tags are then evaluated to select the latest image with tag based on the policy rules, indicated by ImagePolicy.spec.policy.
The ImageUpdateAutomation refers to podinfo GitRepository as the source that should be kept up-to-date, indicated by ImageUpdateAutomation.spec.sourceRef.
The image-automation-controller lists all the ImagePolicies in the ImageUpdateAutomation’s namespace. It then checks out the Git repository main branch, as configured in GitRepository.spec.ref.branch. It then goes through the YAML manifests from the root of the Git repository, as configured in ImageUpdateAutomation.spec.update.path and applies updates based on the latest images from the image policies. The source changes are saved as a Git commit with the commit author defined in ImageUpdateAutomation.spec.git.commit.author. The commit is then push to the remote Git repository’s main branch, indicated by ImageUpdateAutomation.spec.git.push.branch.
The push commit hash is reported in the ImageUpdateAutomation.status.lastPushCommit field and the push time is reported in .status.lastPushTime field.

This example can be run by saving the manifest into imageupdateautomation.yaml.

Apply the resource on the cluster:

kubectl apply -f imageupdateautomation.yaml

Run kubectl get imageupdateautomation to see the ImageUpdateAutomation:

NAME LAST RUN
podinfo-update 2024-03-17T22:22:34Z

Run kubectl describe imageupdateautomation podinfo-update to see the Last Push Commit and Conditions in the ImageUpdateAutomation’s Status:

Status:
 Conditions:
 Last Transition Time: 2024-03-17T22:22:33Z
 Message: repository up-to-date
 Observed Generation: 1
 Reason: Succeeded
 Status: True
 Type: Ready
 Last Automation Run Time: 2024-03-17T22:22:34Z
 Last Push Commit: 3ebb95cc56d2db59bc6ffbe0d9dd0ea445edeb77
 Last Push Time: 2024-03-17T22:22:34Z
 Observed Generation: 1
 Observed Policies:
 Podinfo - Policy:
 Name: ghcr.io/stefanprodan/podinfo
 Tag: 5.0.3
 Observed Source Revision: main@sha1:3ebb95cc56d2db59bc6ffbe0d9dd0ea445edeb77
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Succeeded 5s (x2 over 6s) image-automation-controller repository up-to-date
 Normal Succeeded 5s image-automation-controller pushed commit '3ebb95c' to branch 'main'
Update from image update automation

Writing an ImageUpdateAutomation spec

As with all other Kubernetes config, an ImageUpdateAutomation needs apiVersion, kind, and metadata fields. The name of an ImageUpdateAutomation object must be a valid DNS subdomain name.

An ImageUpdateAutomation also needs a .spec section.

Source reference

.spec.sourceRef is a required field to specify a reference to a source object in the same namespace as the ImageUpdateAutomation or in another namespace. The only supported source kind at the moment is GitRepository, which is used by default if the .spec.sourceRef.kind is not specified. The source reference name is a required field, .spec.sourceRef.name. The source reference namespace is optional, .spec.sourceRef.namespace. If not specified, the source is assumed to be in the same namespace as the ImageUpdateAutomation. The GitRepository must contain the authentication configuration required to check out the source, if any.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 sourceRef:
 name: <gitrepository-name>
 namespace: <gitrepository-namespace>

By default, GitRepository in a different namespace can be referenced. This can be disabled by setting the controller flag --no-cross-namespace-refs.

The timeouts used in the Git operations for an ImageUpdateAutomation is derived from the referenced GitRepository source. GitRepository.spec.timeout can be tuned to adjust the Git operation timeout.

The proxy configurations are also derived from the referenced GitRepository source. GitRepository.spec.proxySecretRef can be used to configure proxy use.

GitRepository Provider

GitRepository can be configured to specify an OIDC provider for authentication using GitRepository.spec.provider field. Image automation controller can be configured to authenticate using the provider as described below.

For a complete guide on how to set up authentication for cloud providers, see the integration docs.

Azure

If the provider is set to azure, make sure the pre-requisites are satisfied. To configure image automation controller to use workload identity,

Create a managed identity to access Azure DevOps. Establish a federated identity credential between the managed identity and the image-automation-controller service account. In the default installation, the image-automation-controller service account is located in the flux-system namespace with name image-automation-controller. Ensure the federated credential uses the correct namespace and name of the image-automation-controller service account. For more details, please refer to this guide.
Add the managed identity to the Azure DevOps organization as a user. Ensure that the managed identity has the necessary permissions to access the Azure DevOps repository as described here.
Add the following patch to your bootstrap repository in flux-system/kustomization.yaml file.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - patch: |-
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: image-automation-controller
 namespace: flux-system
 annotations:
 azure.workload.identity/client-id: <AZURE_CLIENT_ID>
 labels:
 azure.workload.identity/use: "true"
 - patch: |-
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: image-automation-controller
 namespace: flux-system
 labels:
 azure.workload.identity/use: "true"
 spec:
 template:
 metadata:
 labels:
 azure.workload.identity/use: "true"

GitHub

If the provider is set to github, make sure the GitHub App is registered and installed with the necessary permissions and the github app secret is configured as described here.

Git specification

.spec.git is a required field to specify Git configurations related to source checkout, commit and push operations.

Checkout

.spec.git.checkout is an optional field to specify the Git reference to check out. The .spec.git.checkout.ref field is the same as the GitRepository.spec.ref field. It can be used to override the checkout configuration in the referenced GitRepository. Not specifying this reference defaults to the checkout reference of the associated GitRepository.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 git:
 checkout:
 ref:
 branch: <branch-name>

If .spec.git.push is unspecified, .spec.git.checkout will be used as the push branch for any updates.

By default the controller will only do shallow clones, but this can be disabled by starting the controller with flag --feature-gates=GitShallowClone=false.

Commit

.spec.git.commit is a required field to specify the details about the commit made by the automation.

Author

.spec.git.commit.author is a required field to specify the commit author. The author .email is required. The author .name is optional. The name and email are used as the author of the commits made by the automation.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 git:
 commit:
 author:
 email: <author-email>
 name: <author-name>

Signing Key

.spec.git.commit.signingKey is an optional field to specify the signing PGP key to sign the commits with. .secretRef.name refers to a Secret in the same namespace as the ImageUpdateAutomation, containing an ASCII-armored PGP key, in a field named git.asc. If the private key is protected by a passphrase, the passphrase can be specified in the same Secret in a field named passphrase.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 git:
 commit:
 signingKey:
 secretRef:
 name: signing-key
...
---
apiVersion: v1
kind: Secret
metadata:
 name: signing-key
stringData:
 git.asc: |
 <ARMOR ENCODED PGP KEY>
 passphrase: <private-key-passphrase>

Message Template

.spec.git.commit.messageTemplate is an optional field to specify the commit message template. If unspecified, a default commit message is used.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 git:
 commit:
 messageTemplate: |-
 Automated image update by Flux

Removal Note: The Updated template data has been removed from the API. Use Changed template data instead, as it accommodates for all the updates, including partial updates to just the image name or the tag, not just full image with name and tag update. Templates using Updated will result in an error and the ImageUpdateAutomation will be marked as Stalled.

The message template also has access to the data related to the changes made by the automation. The template is a Go text template. The data available to the template have the following structure (not reproduced verbatim):

// TemplateData is the type of the value given to the commit message
// template.
type TemplateData struct {
 AutomationObject struct {
 Name, Namespace string
 }
 Changed update.Result
 Values map[string]string
}

// Result contains the file changes made during the update. It contains
// details about the exact changes made to the files and the objects in them. It
// has a nested structure file->objects->changes.
type Result struct {
 FileChanges map[string]ObjectChanges
}

// ObjectChanges contains all the changes made to objects.
type ObjectChanges map[ObjectIdentifier][]Change

// ObjectIdentifier holds the identifying data for a particular
// object. This won't always have a name (e.g., a kustomization.yaml).
type ObjectIdentifier struct {
 Name, Namespace, APIVersion, Kind string
}

// Change contains the setter that resulted in a Change, the old and the new
// value after the Change.
type Change struct {
 OldValue string
 NewValue string
 Setter string
}

The Changed template data field also has a few helper methods to easily range over the changed objects and changes:

// Changes returns all the changes that were made in at least one update.
func (r Result) Changes() []Change

// Objects returns ObjectChanges, regardless of which file they appear in.
func (r Result) Objects() ObjectChanges

Example of using the methods in a template:

spec:
 commit:
 messageTemplate: |
 Automated image update

 Automation name: {{ .AutomationObject }}

 Files:
 {{ range $filename, $_ := .Changed.FileChanges -}}
 - {{ $filename }}
 {{ end -}}

 Objects:
 {{ range $resource, $changes := .Changed.Objects -}}
 - {{ $resource.Kind }} {{ $resource.Name }}
 Changes:
 {{- range $_, $change := $changes }}
 - {{ $change.OldValue }} -> {{ $change.NewValue }}
 {{ end -}}
 {{ end -}}

With template functions, it is possible to manipulate and transform the supplied data in order to generate more complex commit messages.

spec:
 commit:
 messageTemplate: |
 Automated image update

 Automation name: {{ .AutomationObject }}

 Files:
 {{ range $filename, $_ := .Changed.FileChanges -}}
 - {{ $filename }}
 {{ end -}}

 Objects:
 {{ range $resource, $changes := .Changed.Objects -}}
 - {{ $resource.Kind | lower }} {{ $resource.Name | lower }}
 Changes:
 {{- range $_, $change := $changes }}
 {{ if contains "5.0.3" $change.NewValue -}}
 - {{ $change.OldValue }} -> {{ $change.NewValue }}
 {{ else -}}
 [skip ci] wrong image
 {{ end -}}
 {{ end -}}
 {{ end -}}

There are over 70 available functions. Some of them are defined by the Go template language itself. Most of the others are part of the Sprig template library.

Additional data can be provided with .spec.git.commit.messageTemplateValues.

This is a key/value mapping with string values.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 git:
 commit:
 messageTemplate: |-
 Automated image update by Flux for cluster {{ .Values.cluster }}.
 messageTemplateValues:
 cluster: prod

Push

.spec.git.push is an optional field that specifies how the commits are pushed to the remote source repository.

Branch

.spec.git.push.branch field specifies the remote branch to push to. If unspecified, the commits are pushed to the branch specified in .spec.git.checkout.branch. If .spec.git.checkout is also unspecified, it will fall back to the branch specified in the associated GitRepository’s .spec.sourceRef. If none of these yield a push branch name, the automation will fail.

The push branch will be created locally if it does not already exist, starting from the checkout branch. If the push branch already exists, it will be overwritten with the cloned version plus the changes made by the controller. Alternatively, force push can be disabled by starting the controller with flag --feature-gates=GitForcePushBranch=false, in which case the updates will be calculated on top of any commits already on the push branch. Note that without force push in push branches, if the target branch is stale, the controller may not be able to conclude the operation and will consistently fail until the branch is either deleted or refreshed.

In the following snippet, updates will be pushed as commits to the branch auto, and when that branch does not exist at the origin, it will be created locally starting from the branch main, and pushed:

spec:
 git:
 checkout:
 ref:
 branch: main
 push:
 branch: auto

Refspec

.spec.git.push.refspec field specifies the refspec to push to any arbitrary destination reference. An example of a valid refspec is refs/heads/branch:refs/heads/branch.

If both .push.refspec and .push.branch are specified, then the reconciler will push to both the destinations. This is particularly useful for working with Gerrit servers. For more information about this, please refer to the Gerrit section. This can also be used to automatically open Pull-Requests in Gitea or Forgejo. See the Gitea section for an example.

If only .push.refspec is set, without explicitly defining a .push.branch, the controller falls back to pushing to the branch from checkoutRef and also pushes to .push.refspec.

Note: If both .push.refspec and .push.branch are essentially equal to each other (for e.g.: .push.refspec: refs/heads/main:refs/heads/main and .push.branch: main), then the reconciler might fail with an already up-to-date error.

In the following snippet, updates and commits will be made on the main branch locally. The commits will be then pushed using the refs/heads/main:refs/heads/auto refspec:

spec:
 git:
 checkout:
 ref:
 branch: main
 push:
 refspec: refs/heads/main:refs/heads/auto

Push options

To specify the push options to be sent to the upstream Git server, use .push.options. These options can be used to perform operations as a result of the push. For example, using the below push options will open a GitLab Merge Request to the release branch automatically with the commit the controller pushed to the dev branch:

spec:
 git:
 push:
 branch: dev
 options:
 merge_request.create: ""
 merge_request.target: release

Interval

.spec.interval is a required field that specifies the interval at which the Image update is attempted.

After successfully reconciling the object, the image-automation-controller requeues it for inspection after the specified interval. The value must be in a Go recognized duration string format, e.g. 10m0s to reconcile the object every 10 minutes.

If the .metadata.generation of a resource changes (due to e.g. a change to the spec), this is handled instantly outside the interval window.

Update

.spec.update is an optional field that specifies how to carry out the updates on a source. The only supported update strategy at the moment is Setters, which is used by default for .spec.update.strategy field. The .spec.update.path is an optional field to specify the directory containing the manifests to be updated. If not specified, it defaults to the root of the source repository.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 update:
 path: </path/to/manifest>

Suspend

.spec.suspend is an optional field to suspend the reconciliation of an ImageUpdateAutomation. When set to true, the controller will stop reconciling the ImageUpdateAutomation, and changes to the resource or image policies or Git repository will not result in any update. When the field is set to false or removed, it will resume.

PolicySelector

.spec.policySelector is an optional field to limit policies that an ImageUpdateAutomation takes into account. It supports the same selectors as Deployment.spec.selector (matchLabels and matchExpressions fields). If not specified, it defaults to matchLabels: {} which means all policies in namespace.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 policySelector:
 matchLabels:
 app.kubernetes.io/instance: my-app
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 policySelector:
 matchExpressions:
 - key: app.kubernetes.io/component
 operator: In
 values:
 - my-component
 - my-other-component

Working with ImageUpdateAutomation

Marking images for update

In order to tell ImageUpdateAutomation to update images in a manifest, the images must be marked with setters. A setter is a comment at the end of a line telling exactly which ImagePolicy to use for that line, and optionally also which field of the image elected as latest by the policy to update in that line. This is useful for example in Helm charts, where images are often defined by multiple values, such as the repository URL, a tag, and optionally a digest.

For example, if you want to use the ImagePolicy my-policy from the flux-system namespace, you could use it in a Deployment like this:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-app
 namespace: default
spec:
 template:
 spec:
 containers:
 - name: my-app
 image: ghcr.io/my-org/my-app:4.0.6 # {"$imagepolicy": "flux-system:my-policy"}

Deployments expect the image to be fully specified, including tag and optionally digest, so for Deployments the setter is the basic one, not specifying any fields of the image.

If your app is instead deployed by a Flux HelmRelease whose chart supports image fields, you can use the image fields like this:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-app
 namespace: default
spec:
 values:
 image:
 repository: ghcr.io/my-org/my-app # {"$imagepolicy": "flux-system:my-policy:name"}
 tag: 4.0.6 # {"$imagepolicy": "flux-system:my-policy:tag"}
 digest: sha256:6129bb944cada32b4662eafd13fd9904d34c77286bf2ec4523eaedb711757cb0 # {"$imagepolicy": "flux-system:my-policy:digest"}

Note: For the digest field to be available in the ImagePolicy status, the .spec.digestReflectionPolicy field of the ImagePolicy must be set to IfNotPresent or Always. For a complete guide on digest reflection, see these docs.

Triggering a reconciliation

To manually tell the image-automation-controller to reconcile an ImageUpdateAutomation outside of the specified interval window, an ImageUpdateAutomation can be annotated with reconcile.fluxcd.io/requestedAt: <arbitrary value>. Annotating the resource queues the ImageUpdateAutomation for reconciliation if the <arbitrary-value> differs from the last value the controller acted on, as reported in .status.lastHandledReconcileAt.

Using kubectl:

kubectl annotate --field-manager=flux-client-side-apply --overwrite imageupdateautomation/<automation-name> reconcile.fluxcd.io/requestedAt="$(date +%s)"

Using flux:

flux reconcile image update <automation-name>

Waiting for `Ready`

When a change is applied, it is possible to wait for the ImageUpdateAutomation to reach a ready state using kubectl:

kubectl wait imageupdateautomation/<automation-name> --for=condition=ready --timeout=1m

Suspending and resuming

When you find yourself in a situation where you temporarily want to pause the reconciliation of a ImageUpdateAutomation, you can suspend it using the .spec.suspend field.

Suspend an ImageUpdateAutomation

In your YAML declaration:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 suspend: true

Using kubectl:

kubectl patch imageupdateautomation <automation-name> --field-manager=flux-client-side-apply -p '{\"spec\": {\"suspend\" : true }}'

Using flux:

flux suspend image update <automation-name>

Resume an ImageUpdateAutomation

In your YAML declaration, comment out (or remove) the .spec.suspend field:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 # suspend: true

Using kubectl:

kubectl patch imageupdateautomation <automation-name> --field-manager=flux-client-side-apply -p '{\"spec\" : {\"suspend\" : false }}'

Using flux:

flux resume image update <automation-name>

Debugging an ImageUpdateAutomation

There are several ways to gather information about an ImageUpdateAutomation for debugging purposes.

Describe the ImageUpdateAutomation

Describing an ImageUpdateAutomation using kubectl describe imageupdateautomation <automation-name> displays the latest recorded information for the resource in the Status and Events sections:

...
Status:
 Conditions:
 Last Transition Time: 2024-03-18T20:00:56Z
 Message: processing object: new generation 6 -> 7
 Observed Generation: 7
 Reason: ProgressingWithRetry
 Status: True
 Type: Reconciling
 Last Transition Time: 2024-03-18T20:00:54Z
 Message: failed to checkout source: unable to clone 'https://github.com/fluxcd/example': couldn't find remote ref "refs/heads/non-existing-branch"
 Observed Generation: 7
 Reason: GitOperationFailed
 Status: False
 Type: Ready
 Last Automation Run Time: 2024-03-18T20:00:56Z
 Last Handled Reconcile At: 1710791381
 Last Push Commit: 8084f1bb180ac259c6698cd027064b7dce86a72a
 Last Push Time: 2024-03-18T18:53:04Z
 Observed Generation: 6
 Observed Policies:
 Podinfo - Policy:
 Name: ghcr.io/stefanprodan/podinfo
 Tag: 4.0.6
 Observed Source Revision: main@sha1:8084f1bb180ac259c6698cd027064b7dce86a72a
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Succeeded 11m (x11 over 170m) image-automation-controller no change since last reconciliation
 Warning GitOperationFailed 2s (x3 over 4s) image-automation-controller failed to checkout source: unable to clone 'https://github.com/fluxcd/example': couldn't find remote ref "refs/heads/non-existing-branch"

Trace emitted Events

To view events for specific ImageUpdateAutomation(s), kubectl events can be used in combination with --for to list the Events for specific objects. For example, running

kubectl events --for ImageUpdateAutomation/<automation-name>

lists

LAST SEEN TYPE REASON OBJECT MESSAGE
3m29s (x7 over 4m17s) Warning GitOperationFailed ImageUpdateAutomation/<automation-name> failed to checkout source: unable to clone 'https://github.com/fluxcd/example': couldn't find remote ref "refs/heads/non-existing-branch"
3m14s (x4 over 3h24m) Normal Succeeded ImageUpdateAutomation/<automation-name> repository up-to-date
2m41s (x12 over 174m) Normal Succeeded ImageUpdateAutomation/<automation-name> no change since last reconciliation

Besides being reported in Events, the reconciliation errors are also logged by the controller. The Flux CLI offer commands for filtering the logs for a specific ImageUpdateAutomation, e.g. flux logs --level=error --kind=ImageUpdateAutomation --name=<automation-name>.

Gerrit

Gerrit operates differently from a standard Git server. Rather than sending individual commits to a branch, all changes are bundled into a single commit. This commit requires a distinct identifier separate from the commit SHA. Additionally, instead of initiating a Pull Request between branches, the commit is pushed using a refspec: HEAD:refs/for/main.

As the image-automation-controller is primarily designed to work with standard Git servers, these special characteristics necessitate a few workarounds. The following is an example configuration that works well with Gerrit:

spec:
 git:
 checkout:
 ref:
 branch: main
 commit:
 author:
 email: flux@localdomain
 name: flux
 messageTemplate: |
 Perform automatic image update

 Automation name: {{ .AutomationObject }}

 {{- $ChangeId := .AutomationObject -}}
 {{- $ChangeId = printf "%s%s" $ChangeId ( .Changed.FileChanges | toString ) -}}
 {{- $ChangeId = printf "%s%s" $ChangeId ( .Changed.Objects | toString ) -}}
 {{- $ChangeId = printf "%s%s" $ChangeId ( .Changed.Changes | toString ) }}
 Change-Id: {{ printf "I%s" ( sha256sum $ChangeId | trunc 40 ) }}
 push:
 branch: auto
 refspec: refs/heads/auto:refs/heads/main

This instructs the image-automation-controller to clone the repository using the main branch but execute its update logic and commit with the provided message template on the auto branch. Commits are then pushed to the auto branch, followed by pushing the HEAD of the auto branch to the HEAD of the remote main branch. The message template ensures the inclusion of a Change-Id at the bottom of the commit message.

The initial branch push aims to prevent multiple Patch Sets. If we exclude .push.branch and only specify .push.refspec: refs/heads/main:refs/heads/main, the desired Change can be created as intended. However, when the controller freshly clones the main branch while a Change is open, it executes its update logic on main, leading to new commits being pushed with the same changes to the existing open Change. Specifying .push.branch circumvents this by instructing the controller to apply the update logic to the auto branch, already containing the desired commit. This approach is also recommended in the Gerrit documentation.

Another thing to note is the syntax of .push.refspec. Instead of it being HEAD:refs/for/main, commonly used by Gerrit users, we specify the full refname refs/heads/auto in the source part of the refpsec.

Note: A known limitation of using the image-automation-controller with Gerrit involves handling multiple concurrent Changes. This is due to the calculation of the Change-Id, relying on factors like file names and image tags. If the controller introduces a new file or modifies a previously updated image tag to a different one, it leads to a distinct Change-Id for the commit. Consequently, this action will trigger the creation of an additional Change, even when an existing Change containing outdated modifications remains open.

Gitea

Gitea and Forgejo each implement the AGit-Workflow. This means, the image-automation-controller is able to open a pull-request by pushing to a refspec like HEAD:refs/for/main with the apropriate push-options.

The following example opens a PR on a Gitea or Forgejo-Server:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: my-automation
 namespace: flux-system
spec:
 git:
 checkout:
 ref:
 branch: main
 push:
 branch: flux-updates
 refspec: refs/heads/flux-updates:refs/for/main
 options:
 topic: flux-updates
 title: Flux Image Update
 description: |-
 This PR is automatically opened by the fluxcd *image-automation-controller*
 commit:
 author:
 email: flux@eaample.com
 name: fluxcd
 messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println
 .NewValue}}{{end}}'

ImageUpdateAutomation Status

Observed Policies

The ImageUpdateAutomation reports the observed image policies that were considered during the image update in the .status.observedPolicies field. It is a map of the policy name and its latest image name and tag.

Example:

status:
 ...
 observedPolicies:
 podinfo-policy:
 name: ghcr.io/stefanprodan/podinfo
 tag: 4.0.6
 myapp1:
 name: ghcr.io/fluxcd/myapp1
 tag: 4.0.0
 myapp2:
 name: ghcr.io/fluxcd/myapp2
 tag: 2.0.0
 ...

The observed policies keep track of the policies considered in the last reconciliation and is used to determine if the reconciliation can skip full execution due to no change in image policies or remote source.

Observed Source Revision

The ImageUpdateAutomation reports the observed source revision that was checked out during the image update in the .status.observedSourceRevision field. For a GitRepository, the observed source revision would contain the branch name and the commit hash; e.g., main@sha1:8084f1bb180ac259c6698cd027064b7dce86a72a. If the checkout and push branchs are the same, the commit hash of the observed source revision is equal to the last push commit.

The observed source revision keeps track of the source revision seen in the last reconciliation and is used to determine if the reconciliation can skip full execution due to no change in image policies or remote source.

Last Automation Run Time

The ImageUpdateAutomation reports the last automation run time in the .status.lastAutomationRunTime field. It is a timestamp of when the reconciliation ran the last time, regardless of any effective resulting update.

Last Push Commit

The ImageUpdateAutomation reports the last pushed commit for image update in the .status.lastPushCommit field. It is the commit hash of the last pushed commit. The commit has may not be the same that’s present in the observed source revision if the puch branch is different from the checkout branch or the remote repository has new commits which didn’t result in an image update.

Last Push Time

The ImageUpdateAutomation reports the last pushed commit time for image update in the .status.lastPushTime field. It is a timestamp of when the last image update resulted in a pushing of new commit to the source.

Conditions

An ImageUpdateAutomation enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be reconciling while checking out and updating images in source, it can be ready, or it can fail during reconciliation.

The ImageUpdateAutomation API is compatible with the kstatus specification, and reports Reconciling and Stalled conditions where applicable to provide better (timeout) support to solutions polling the ImageUpdateAutomation to become Ready.

Reconciling ImageUpdateAutomation

The image-automation-controller marks an ImageUpdateAutomation as reconciling when one of the following is true:

The generation of the ImageUpdateAutomation is newer than the Observed Generation.
The ImageUpdateAutomation has observed new ImagePolicies or changes in the ImagePolicies’ latest images, or change in the remote source.

When the ImageUpdateAutomation is “reconciling”, the Ready Condition status becomes Unknown, and the controller adds a Condition with the following attributes to the ImageUpdateAutomation’s .status.conditions:

type: Reconciling
status: "True"
reason: Progressing

It has a “negative polarity”, and is only present on the ImageUpdateAutomation while its status value is "True".

Ready ImageUpdateAutomation

The image-automation-controller marks an ImageUpdateAutomation as ready when it has the following characteristics:

The controller was able to check out the remote source repository using the specified GitRepository configurations.
The ImageUpdateAutomation could not find any update to the source, already up-to-date.
The ImageUpdateAutomation pushes image updates to the source, making it up-to-date.

When the ImageUpdateAutomation is “ready”, the controller sets a Condition with the following attributes in the ImageUpdateAutomation’s .status.conditions:

type: Ready
status: "True"
reason: Succeeded

This Ready Condition will retain a status value of "True" until a failure occurs due to any reason.

Failed ImageUpdateAutomation

The image-automation-controller may get stuck trying to update a source without completing. This can occur due to some of the following factors:

The remote source is temporarily unavailable.
The referenced source is in a different namespace and cross-namespace reference is disabled.
The referenced source does not exist.
The credentials associated with the source are invalid.
The source configuration is invalid for the current state of the source, for example, the specified branch does not exists in the remote source repository.
The remote source repository prevents push or creation of new push branch.
The policy selector is invalid, for example, label is too long.

When this happens, the controller sets the Ready Condition status to False with the following reasons:

reason: AccessDenied | reason: InvalidSourceConfiguration | reason: GitOperationFailed | reason: UpdateFailed | reason: InvalidPolicySelector

While the ImageUpdateAutomation is in failing state, the controller will continue to attempt to update the source with an exponential backoff, until it succeeds and the ImageUpdateAutomation is marked as ready.

Note that an ImageUpdateAutomation can be reconciling while failing at the same time, for example due to a newly introduced configuration issue in the ImageUpdateAutomation spec.

Observed Generation

The image-automation-controller reports an observed generation in the ImageUpdateAutomation’s .status.observedGeneration. The observed generation is the latest .metadata.generation which resulted in either a ready state, or stalled due to error it can not recover from without human intervention.

Last Handled Reconcile At

The image-automation-controller reports the last reconcile.fluxcd.io/requestedAt annotation value it acted on in the .status.lastHandledReconcileAt field.

For practical information about this field, see triggering a reconcile.

Flux – Image reflector and automation controllers

Flux: Controller Options

Image automation flags

Feature Gates

Image reflector flags

Feature Gates

Flux: Image Policies

Example

Writing an ImagePolicy spec

Image Repository Reference

Policy

SemVer

Alphabetical

Numerical

Filter Tags

Digest Reflection

Interval

Working with ImagePolicy

Triggering a reconcile

Waiting for Ready

Debugging an ImagePolicy

Describe the ImagePolicy

Trace emitted Events

ImagePolicy Status

Latest Ref

Observed Previous Ref

Conditions

Reconciling ImagePolicy

Ready ImagePolicy

Failed ImagePolicy

Observed Generation

Flux: Image Repositories

Example

Writing an ImageRepository spec

Image

Interval

Timeout

Secret reference

ServiceAccount name

Certificate secret reference

Supported configurations

Mutual TLS Authentication

Example: mTLS Configuration

CA Certificate Authentication

Example: CA Certificate Configuration

Proxy secret reference

Suspend

Access from

Exclusion list

Insecure

Provider

AWS

Worker Node IAM

IAM roles for service accounts(IRSA)

Azure

Kubelet Identity

Workload Identity

GCP

Access scopes

Workload Identity

Authentication on other platforms

Working with ImageRepositories

Triggering a reconcile

Waiting for Ready

Suspending and resuming

Suspend an ImageRepository

Resume an ImageRepository

Debugging an ImageRepository

Describe the ImageRepository

Trace emitted Events

ImageRepository Status

Last Scan Result

Canonical Image Name

Observed Exclusion List

Conditions

Reconciling ImageRepository

Ready ImageRepository

Failed ImageRepository

Observed Generation

Last Handled Reconcile At

Waiting for `Ready`

Waiting for `Ready`

Waiting for `Ready`