Flux – Integrations

Flux: Amazon Web Services

Mon, 01 Jan 0001 00:00:00 +0000

The Flux APIs integrate with the following Amazon Web Services (AWS) services:

The source-controller integrates the OCIRepository API with Amazon Elastic Container Registry (ECR) for pulling OCI artifacts into the cluster.
The source-controller integrates the OCIRepository API with Amazon Public Elastic Container Registry (Public ECR) for pulling OCI artifacts into the cluster.
The image-reflector-controller integrates the ImageRepository and ImagePolicy APIs with ECR and public ECR for scanning tags and digests of OCI artifacts and reflecting them into the cluster.
The source-controller integrates the Bucket API with Amazon Simple Storage Service (S3) for pulling manifests from buckets and packaging them as artifacts inside the cluster.
The kustomize-controller integrates the Kustomization API with Amazon Key Management Service (KMS) for decrypting SOPS-encrypted secrets when applying manifests in the cluster.
The kustomize-controller integrates the Kustomization API with Amazon Elastic Kubernetes Service (EKS) for applying manifests in remote EKS clusters.
The helm-controller integrates the HelmRelease API with EKS for applying Helm charts in remote EKS clusters.

The next sections briefly describe AWS IAM, AWS’s identity and access management system, and how it can be used to grant Flux access to resources offered by the services above. Bear in mind that AWS IAM has more features than the ones described here. We describe only the features that are relevant for the Flux integrations.

Identity

For all the integrations with AWS, AWS will authenticate two types of identities for Flux:

Each has its own methods of authentication. In particular, IAM Roles support secret-less authentication, while IAM Users support secret-based authentication. The former is more secure.

For further understanding how to configure authentication for both types of identities, refer to the Authentication section.

Access Management

Identities in AWS need permissions to access resources from AWS services. Those can be granted using permission policies. Policies group statements that allow specific actions to be performed on specific resources of specific services. Each policy has a name and a globally unique Amazon Resource Name (ARN). For example, the policy AmazonS3ReadOnlyAccess, whose ARN is arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess, groups the following statements:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:Get*",
 "s3:List*",
 "s3:Describe*",
 "s3-object-lambda:Get*",
 "s3-object-lambda:List*"
 ],
 "Resource": "*"
 }
 ]
}

The structure of actions is usually <service>:<action>.

AWS has a large number of AWS-managed policies, but users can also create customer-managed policies inside their AWS accounts.

AWS supports also inline policies.

Managed and inline policies can be attached to identities, which is a good strategy for when the identity and the resource are in the same AWS account.

Only inline policies can be attached to resources. This is required when the target resource is in a different AWS account than the identity.

Note: As an exception, KMS does not support identity-based policies by default. To enable it, follow these docs.

Granting permissions

When attaching permission policies to identities, a statement must enumerate the resources it applies to. This can be done using ARNs, but it can also be done using wildcards. For example, the AmazonS3ReadOnlyAccess policy above uses the wildcard * to allow access to all S3 resources. This is not recommended, as it violates the Least Privilege Principle. The recommended way of granting permissions to Flux is to use specific ARNs, for example:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "AllowReadObjects",
 "Effect": "Allow",
 "Action": [
 "s3:GetObject",
 "s3:ListBucket"
 ],
 "Resource": [
 "arn:aws:s3:::flux-bucket",
 "arn:aws:s3:::flux-bucket/*"
 ]
 }
 ]
}

When attaching (inline) permission policies to resources, a statement must enumerate also the identities it applies to, through identity ARNs. For example:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "AllowReadObjects",
 "Effect": "Allow",
 "Principal": {
 "AWS": [
 "arn:aws:iam::123456789012:user/some-user",
 "arn:aws:iam::123456789012:role/some-role"
 ]
 },
 "Action": [
 "s3:GetObject",
 "s3:ListBucket"
 ],
 "Resource": [
 "arn:aws:s3:::flux-bucket",
 "arn:aws:s3:::flux-bucket/*"
 ]
 }
 ]
}

Via AWS Controllers for Kubernetes IAM custom resources

To use AWS Controllers for Kubernetes (ACK) for creating permission policies, you can use the following custom resource:

Policy

Then to create IAM Roles and Users and attach the policies to them, you can use the following:

For resource-based policies, see the Authorization section below for each individual resource type that Flux integrates with.

Note: ACK is the most GitOps-friendly way of managing AWS resources. With it you can even use Flux itself to deploy and continuously reconcile your AWS custom resources (the Kubernetes ones), and ACK itself will continuously reconcile the actual AWS resources.

Via Terraform/OpenTofu IAM resources

To use Terraform/OpenTofu for creating permission policies, you can use the following resource:

aws_iam_policy

Then to create IAM Roles and Users and attach the policies to them, you can use the following:

For resource-based policies, see the Authorization section below for each individual resource type that Flux integrates with.

Note: Terraform/OpenTofu is a great declarative way of managing AWS resources, but it’s not as GitOps-friendly as ACK, as these tools do not prescribe continuous reconciliation of the resources. There are alternatives to achieve that, but they are not a core part of the tools.

Via the `aws` CLI

To use the aws CLI for creating permission policies you can use the following command:

aws iam create-policy

Then to create IAM Roles and Users and attach the policies to them, you can use the following:

For resource-based policies, see the Authorization section below for each individual resource type that Flux integrates with.

Note: The aws CLI is a great tool for experimenting with AWS resources and their integrations with Flux, but it’s not a GitOps-friendly way of managing AWS resources. If you need continuous reconciliation of the resources, prefer using ACK or Terraform/OpenTofu.

Allowing Kubernetes Service Accounts to assume IAM Roles (Trust Policies)

For EKS Pod Identity and OIDC Federation, the only supported identity type is IAM Roles. In both cases, an IAM Role must be configured to allow a Kubernetes Service Account to assume it. This is done by adding a Trust Policy to the IAM Role.

Trust policies are similar in syntax to permission policies, but for each of these two features the trust policy JSON document has particular requirements. For EKS Pod Identity, see here. For OIDC Federation, see here.

To create an IAM Role with a trust policy using ACK, you can use the following custom resource (look for the spec.assumeRolePolicyDocument field):

Role

To create an IAM Role with a trust policy using Terraform/OpenTofu, you can use the following resource (look for the assume_role_policy argument):

aws_iam_role

To configure a trust policy for an IAM Role using the aws CLI, you can use the following command:

aws iam update-assume-role-policy

Authorization

In this section we describe the recommended IAM policy bindings for enabling the Flux integrations with AWS services.

For Amazon Elastic Container Registry

The OCIRepository, ImageRepository and ImagePolicy Flux APIs are integrated with ECR. The OCIRepository API can be used to pull OCI artifacts from ECR repositories into the cluster, while the ImageRepository and ImagePolicy APIs can be used to reflect tags and digests of such artifacts also inside the cluster.

For single-tenant setups, the recommended AWS-managed policy containing the required permissions for the OCIRepository, ImageRepository and ImagePolicy APIs is:

AmazonEC2ContainerRegistryReadOnly (arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly)

It can be attached to IAM Roles or Users.

This policy grants read-only access to all resources ("Resource": "*"). Hence, for multi-tenant setups, a better approach is attaching an inline policy to the ECR repositories belonging to the tenant granting access to an IAM Role or User with the following permissions:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::123456789012:role/some-tenant-role"
 },
 "Action": [
 "ecr:GetAuthorizationToken",
 "ecr:BatchCheckLayerAvailability",
 "ecr:GetDownloadUrlForLayer",
 "ecr:GetRepositoryPolicy",
 "ecr:DescribeRepositories",
 "ecr:ListImages",
 "ecr:DescribeImages",
 "ecr:BatchGetImage",
 "ecr:GetLifecyclePolicy",
 "ecr:GetLifecyclePolicyPreview",
 "ecr:ListTagsForResource",
 "ecr:DescribeImageScanFindings"
 ],
 "Resource": "*"
 }
 ]
}

The ACK resource kind for creating an ECR repository and attaching an inline permission policy is:

Repository

The Terraform/OpenTofu resource for attaching an inline permission policy to an ECR repository is:

aws_ecr_repository_policy

The aws CLI command for attaching an inline permission policy to an ECR repository is:

aws ecr set-repository-policy

For Amazon Public Elastic Container Registry

The OCIRepository, ImageRepository and ImagePolicy Flux APIs are integrated with public ECR. The OCIRepository API can be used to pull OCI artifacts from public ECR repositories into the cluster, while the ImageRepository and ImagePolicy APIs can be used to reflect tags and digests of such artifacts also inside the cluster.

For public ECR, attaching the read-only AWS-managed policy suffices for any of the Flux APIs:

AmazonElasticContainerRegistryPublicReadOnly (arn:aws:iam::aws:policy/AmazonElasticContainerRegistryPublicReadOnly)

It can be attached to IAM Roles or Users.

For Amazon Simple Storage Service

The Bucket Flux API is integrated with S3. The Bucket API can be used to pull manifests from S3 buckets and package them as artifacts inside the cluster.

For single-tenant setups, the recommended AWS-managed policy containing the required permissions for the Bucket API is:

AmazonS3ReadOnlyAccess (arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess)

It can be attached to IAM Roles or Users.

This policy grants read-only access to all resources ("Resource": "*"). Hence, for multi-tenant setups, a better approach is attaching an inline policy to the S3 buckets belonging to the tenant granting access to an IAM Role or User with the following permissions:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::123456789012:role/some-tenant-role"
 },
 "Action": [
 "s3:Get*",
 "s3:List*",
 "s3:Describe*",
 "s3-object-lambda:Get*",
 "s3-object-lambda:List*"
 ],
 "Resource": "*"
 }
 ]
}

The ACK resource kind for creating an S3 bucket and attaching an inline permission policy is:

Bucket

The Terraform/OpenTofu resource for attaching an inline permission policy to an S3 bucket is:

aws_s3_bucket_policy

The aws CLI command for attaching an inline permission policy to an S3 bucket is:

aws s3api put-bucket-policy

For Amazon Key Management Service

The Kustomization Flux API is integrated with KMS. The Kustomization API is used to apply manifests in the cluster, and it can use KMS to decrypt SOPS-encrypted secrets before applying them.

As mentioned in the Access Management section, KMS does not support identity-based policies by default, and it’s also part of this security strategy not to offer AWS-managed policies for KMS.

The recommended approach for granting an IAM Role or User access to a KMS key is to attach an inline policy to the KMS key itself, granting access to the IAM Role or User with the following permissions:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::123456789012:role/some-tenant-role"
 },
 "Action": [
 "kms:Decrypt",
 "kms:DescribeKey"
 ],
 "Resource": "*"
 }
 ]
}

A similar policy can be used to grant access to specific KMS keys to IAM Roles or Users with a customer-managed policy.

The ACK resource kind for creating a KMS key and attaching an inline permission policy is:

Key

The Terraform/OpenTofu resource for attaching an inline permission policy to a KMS key is:

aws_kms_key_policy

The aws CLI command for attaching an inline permission policy to a KMS key is:

aws kms put-key-policy

For Amazon Elastic Kubernetes Service

The Kustomization and HelmRelease Flux APIs can use IAM Roles or Users for applying and managing resources in remote EKS clusters. Two kinds of access must be configured for this:

The IAM Role or User must have permission to call the DescribeCluster EKS API for the target remote cluster. The Flux controllers need to call this API for retrieving details required for connecting to the remote cluster, like the cluster’s API server endpoint and certificate authority data. This is done by attaching a permission policy to the IAM Role or User that allows the eks:DescribeCluster action on the target remote cluster.
The IAM Role or User must have permissions inside the remote cluster to apply and manage the target resources. For this, there must exist an Access Entry that maps the IAM Role or User to a Kubernetes username and set of groups that will be used for Kubernetes RBAC inside the remote cluster. Optionally, the Access Entry can also have Access Policies, which are predefined sets of Kubernetes permissions defined by EKS. The resulting set of permissions granted to the IAM Role or User will be the union of: 1) the permissions granted by the Access Policies; with 2) the permissions granted via Kubernetes RBAC through the username and groups configured in the Access Entry, by referencing them in RoleBinding or ClusterRoleBinding objects inside the cluster.

Permissions for the EKS API

The following permission policy can be used for granting access to the DescribeCluster EKS API for a target remote cluster:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::123456789012:role/some-tenant-role"
 },
 "Action": [
 "eks:DescribeCluster"
 ],
 "Resource": "arn:aws:eks:us-east-1:123456789012:cluster/some-cluster"
 }
 ]
}

It can be attached to IAM Roles or Users.

EKS clusters do not support resource-based permission policies.

Permissions inside the remote cluster

The ACK resource kind for creating an EKS Access Entry with associated Access Policies is:

AccessEntry

The Terraform/OpenTofu resources for creating EKS Access Entries and Access Policies are:

The aws CLI command for creating EKS Access Entries and Access Policies are:

For granting Kubernetes RBAC to the Kubernetes username and groups configured in the Access Entry, simply create the corresponding RoleBinding or ClusterRoleBinding objects inside the remote cluster.

Authentication

As mentioned in the Identity section, AWS supports two types of identities for Flux: IAM Roles and IAM Users. This section describes how to authenticate each type of identity. IAM Roles are the recommended way of authenticating to AWS services, as they support secret-less authentication. IAM Users are not recommended, as they require secret-based authentication, which is less secure.

Recommendation: Always prefer secret-less over secret-based authentication if the alternative is available. Secrets can be stolen to abuse the permissions granted to the identities they represent, and for public clouds like AWS this can be done by simply having Internet access. This requires secrets to be regularly rotated and more security controls to be put in place, like audit logs, secret management tools, etc. Secret-less authentication does not have this problem, as the identity is authenticated using a token that is not stored anywhere and is only valid for a short period of time, usually one hour. It’s much harder to steal an identity this way.

With EKS Pod Identity

EKS Pod Identity can be used to link the Kubernetes Service Account of a Flux controller to a single IAM Role that will be used for all the AWS operations this Flux controller performs. This EKS feature can only be used at the controller level.

Note: Pod Identity is an EKS-only feature specifically targeting pods. It’s not possible to support it at the object level, as for a Kubernetes Service Account token to be accepted by Pod Identity it necessarily needs to be tied to a pod that is configured to use this Kubernetes Service Account. This is a guarantee that no Kubernetes controller can provide when issuing a token for a Service Account configured in a custom resource object.

First, enable Pod Identity in the EKS cluster.

In EKS Auto Mode, Pod Identity is always enabled, no cluster setup is required. In standard EKS, follow these docs.

Next, to link an IAM Role with a Kubernetes Service Account of a Flux controller in Pod Identity, the following steps are required:

Allow EKS Pod Identity to assume the IAM Role. This is done by adding a trust policy like this to the IAM Role:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
 "Effect": "Allow",
 "Principal": {
 "Service": "pods.eks.amazonaws.com"
 },
 "Action": [
 "sts:AssumeRole",
 "sts:TagSession"
 ]
 }
 ]
}

To create/attach trust policies to IAM Roles, see this section.

Link the IAM Role to the Kubernetes Service Account in Pod Identity.

To perform this step using ACK, you can use the following custom resource:

PodIdentityAssociation

To perform this step using Terraform/OpenTofu, you can use the following resource:

aws_eks_pod_identity_association

To perform this step using the aws CLI, you can use the following command:

aws eks create-pod-identity-association

Restart (delete) the controller pod for the binding to take effect.

Finally, set the provider field of the Flux resources as described in the controller level section.

With OIDC Federation

OIDC Federation is an AWS feature that allows external identities to authenticate with AWS services without the need for a per-identity static credential. This is done by exchanging a short-lived token issued by the external identity provider (in this case, Kubernetes) for short-lived AWS credentials. These credentials are then used to authenticate with AWS services. OIDC Federation is supported only for IAM Roles.

Supported clusters

AWS supports OIDC Federation for EKS clusters through a feature called IAM Roles for Service Accounts (IRSA), and also for non-EKS clusters.

In EKS clusters you need to create an OIDC Provider with the Issuer URL of the cluster following these docs.

In non-EKS clusters you need to create an OIDC Provider with the Issuer URL of the cluster following these docs.

To create an OIDC Provider using ACK, you can use the following custom resource:

OpenIDConnectProvider

To create an OIDC Provider using Terraform/OpenTofu, you can use the following resource:

aws_iam_openid_connect_provider

If using this Terraform/OpenTofu resource for EKS clusters, you can fetch the Issuer URL of the cluster using this data source (look for the identity.oidc.issuer attribute):

aws_eks_cluster

To create an OIDC Provider using the aws CLI, you can use the following command:

aws iam create-open-id-connect-provider

Supported identity types

As mentioned before, the only identity type supported by OIDC Federation is IAM Roles.

Flux acquires the permission policies granted to an IAM Role by using a Kubernetes Service Account to assume this IAM Role. We call this process impersonation. To configure a Kubernetes Service Account to assume an IAM Role, two steps are required:

Allow the Kubernetes Service Account to assume the IAM Role. This is done by adding a trust policy to the IAM Role that allows the Kubernetes Service Account to assume it.

An IAM Role trust policy for OIDC federation with a Kubernetes cluster looks like this:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/ISSUER_URL_WITHOUT_SCHEME"
 },
 "Action": "sts:AssumeRoleWithWebIdentity",
 "Condition": {
 "StringEquals": {
 "ISSUER_URL_WITHOUT_SCHEME:aud": "sts.amazonaws.com",
 "ISSUER_URL_WITHOUT_SCHEME:sub": "system:serviceaccount:KSA_NAMESPACE:KSA_NAME"
 }
 }
 }
 ]
}

See here how to create/attach trust policies to IAM Roles.

Annotate the Kubernetes Service Account with the IAM Role ARN. This is done by adding the annotation eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME to the Kubernetes Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: KSA_NAME
 namespace: NAMESPACE
 annotations:
 eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME

Configuring Flux to use a Kubernetes Service Account to authenticate with AWS can be done either at the object level or at the controller level.

With IAM User Access Keys

All AWS integrations except for ECR and public ECR support configuring authentication through an IAM User Access Key.

IAM Users support static credentials that can be used to get access to AWS services called Access Keys. An Access Key consists of an Access Key ID and a Secret Access Key. The Access Key ID is a public identifier for the Access Key, while the Secret Access Key is a secret used to sign requests to AWS services that have a short validity period.

ACK does not support creating Access Keys for IAM Users (see issue).

To create an Access Key for an IAM User using Terraform/OpenTofu, you can use the following resource:

aws_iam_access_key

To create an Access Key for an IAM User using the aws CLI, you can use the following command:

aws iam create-access-key

Configuring Flux to use an IAM User Access Key can be done either at the object level or at the controller level.

At the object level

All the Flux APIs support configuring authentication at the object level. This allows users to adhere to the Least Privilege Principle, as each Flux resource can be configured with its own identity and therefore its own permissions. This is useful for multi-tenancy scenarios, where different teams can use the same cluster but need to be isolated from each other inside their own namespaces.

For OIDC Federation

Before following the steps below, make sure to complete the cluster setup described here, and to configure the Kubernetes Service Account as described here.

For configuring authentication through a Kubernetes Service Account at the object level the following steps are required:

Enable the feature gate ObjectLevelWorkloadIdentity in the target Flux controller Deployment during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: (some-controller)
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --feature-gates=ObjectLevelWorkloadIdentity=true

Set the .spec.provider field to aws in the Flux resource. For SOPS decryption with the Kustomization API, this field is not required/does not exist, SOPS detects the provider from the metadata in the SOPS-encrypted Secret. For remote cluster access with the Kustomization and HelmRelease APIs the field is .data.provider inside the referenced ConfigMap from .spec.kubeConfig.configMapRef.
Use the .spec.serviceAccountName field to specify the name of the Kubernetes Service Account in the same namespace as the Flux resource. For SOPS decryption with the Kustomization API, the field is .spec.decryption.serviceAccountName. For remote cluster access with the Kustomization and HelmRelease APIs the field is .data.serviceAccountName inside the referenced ConfigMap from .spec.kubeConfig.configMapRef.

Note: The eks.amazonaws.com/role-arn annotation is defined by EKS, but Flux also uses it to identify the IAM Role to assume in non-EKS clusters. This is for providing users with a seamless experience.

For IAM User Access Keys

Only the S3 and KMS integrations support configuring authentication through an IAM User Access Key at the object level.

For configuring authentication through an IAM User Access Key, the .spec.secretRef.name field must be set to the name of the Kubernetes Secret in the same namespace as the Flux resource containing the IAM User Access Key. In the case of the Kustomization API, the field is .spec.decryption.secretRef.name.

The provider field and the key inside the .data field of the Secret depend on the specific Flux API:

For the Bucket API the .spec.provider field must be set to aws, and the keys inside the .data field of the Secret must be accesskey and secretkey.
For SOPS decryption with the Kustomization API, there’s no provider field as SOPS detects the provider from the metadata in the SOPS-encrypted Secret. The key inside the .data field of the Secret must be sops.aws-kms and the value should look like this:

apiVersion: v1
kind: Secret
metadata:
 name: SECRET_NAME
 namespace: NAMESPACE
type: Opaque
data:
 sops.aws-kms: |
 aws_access_key_id: some-access-key-id
 aws_secret_access_key: some-aws-secret-access-key
 aws_session_token: some-aws-session-token # this field is optional

At the controller level

All the Flux APIs support configuring authentication at the controller level. This is more appropriate for single-tenant scenarios, where all the Flux resources inside the cluster belong to the same team and hence can share the same identity and permissions.

At the controller level, regardless if authenticating with EKS Pod Identity, OIDC Federation or IAM User Access Keys, all Flux resources must have the provider field set to aws according to the rules below.

For all Flux resource kinds except for Kustomization and HelmRelease, set the .spec.provider field to aws and leave .spec.serviceAccountName unset.

For SOPS decryption with the Kustomization API, leave the .spec.decryption.serviceAccountName field unset. There’s no provider field for SOPS decryption.

For remote cluster access with the Kustomization and HelmRelease APIs, set the .data.provider field of the ConfigMap referenced by the .spec.kubeConfig.configMapRef field to aws and leave the .data.serviceAccountName field unset.

The controller-level configuration is described in the following sections.

For OIDC Federation

Before following the steps below, make sure to complete the cluster setup described here, and to configure the Kubernetes Service Account as described here.

If the cluster is EKS, the Kubernetes Service Account of the controller must be configured to assume an IAM Role. This is done by adding the eks.amazonaws.com/role-arn annotation to the controller Service Account during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: ServiceAccount
 name: "(some-controller)"
 patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME

If the configuration above is done after bootstrap, restart (delete) the controller for the binding to take effect.

If the cluster is not EKS, the controller Deployment must be patched during bootstrap:

A projected volume must be mounted in the controller Deployment with a Kubernetes Service Account token whose audience is set to sts.amazonaws.com.
The environment variables required by the AWS Go SDK v2 to authenticate through OIDC Federation must be set in the controller Deployment. See an exhaustive list of the environment variables here.

The controller patch should look like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: "(some-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AWS_REGION
 value: AWS_REGION # This is the region of the AWS STS endpoint.
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AWS_ROLE_ARN
 value: arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AWS_ROLE_SESSION_NAME
 value: some-controller.flux-system.AWS_REGION.fluxcd.io
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AWS_WEB_IDENTITY_TOKEN_FILE
 value: /var/run/service-account/aws-token
 - op: add
 path: /spec/template/spec/volumes/-
 value:
 name: aws-token
 projected:
 sources:
 - serviceAccountToken:
 audience: sts.amazonaws.com
 expirationSeconds: 3600
 path: aws-token
 - op: add
 path: /spec/template/spec/containers/0/volumeMounts/-
 value:
 name: aws-token
 mountPath: /var/run/service-account
 readOnly: true

For IAM User Access Keys

Mount the Kubernetes Secret containing the IAM User Access Key and Secret as the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the controller Deployment during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: "(some-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AWS_ACCESS_KEY_ID
 valueFrom:
 secretKeyRef:
 name: some-controller-aws-access-key
 key: accesskey
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AWS_SECRET_ACCESS_KEY
 valueFrom:
 secretKeyRef:
 name: some-controller-aws-access-key
 key: secretkey

The Secret should look like this:

apiVersion: v1
kind: Secret
metadata:
 name: some-controller-aws-access-key
 namespace: flux-system
type: Opaque
stringData:
 accesskey: some-access-key-id
 secretkey: some-aws-secret-access-key

At the node level

Only for the ECR integrations Flux supports authentication at the node level for EKS. This is because users often already have to configure authentication at the node level for EKS to be able to pull container images from ECR in order to start pods. By supporting this authentication method Flux allows users to configure ECR authentication in a single way. See how to create IAM Roles for EKS worker nodes and how to allow them to pull images from ECR.

⚠️ Node level authentication may work for other integrations as well, but Flux only has continuous integration tests for the ECR integration in order to support the specific use case described above.

For node-level authentication to work, the .spec.provider field of the Flux resources must be set to aws.

Flux: Google Cloud Platform

Mon, 01 Jan 0001 00:00:00 +0000

The Flux APIs integrate with the following Google Cloud Platform (GCP) services:

The source-controller integrates the OCIRepository API with Google Cloud Artifact Registry (GAR) for pulling OCI artifacts into the cluster.
The image-reflector-controller integrates the ImageRepository and ImagePolicy APIs with GAR for scanning tags and digests of OCI artifacts and reflecting them into the cluster.
The source-controller integrates the Bucket API with Google Cloud Storage (GCS) for pulling manifests from buckets and packaging them as artifacts inside the cluster.
The kustomize-controller integrates the Kustomization API with Google Cloud Key Management Service (KMS) for decrypting SOPS-encrypted secrets when applying manifests in the cluster.
The kustomize-controller integrates the Kustomization API with Google Kubernetes Engine (GKE) for applying manifests in remote GKE clusters.
The helm-controller integrates the HelmRelease API with GKE for applying Helm charts in remote GKE clusters.
The notification-controller integrates the Provider API with Google Cloud Pub/Sub for sending notifications about Flux resources outside the cluster.

The next sections briefly describe GCP IAM, GCP’s identity and access management system, and how it can be used to grant Flux access to resources offered by the services above. Bear in mind that GCP IAM has more features than the ones described here. We describe only the features that are relevant for the Flux integrations.

Note: Flux also supports Google Chat webhooks, but Google Chat webhooks do not support GCP IAM. For more information about the notification-controller integration with Google Chat, see the Provider API docs.

Identity

For all the integrations with GCP, GCP will authenticate two types of identities for Flux:

Each has its own methods of authentication. In particular, GCP Service Accounts support both secret-based and secret-less authentication. Kubernetes Service Accounts support only secret-less authentication. The latter is more secure and easier to configure, but has a few limitations (a small number of GCP services and edge cases do not have support for it, see the list here).

For further understanding how to configure authentication for both types of identities, refer to the Authentication section.

Access Management

Identities in GCP need permissions to access resources from GCP services. Permissions cannot be granted directly to identities. Instead, GCP uses roles to group permissions, and only roles can be granted to identities. Each role has a title and a globally unique ID. For example, the role Storage Object Viewer, whose ID is roles/storage.objectViewer, groups the following permissions (most omitted for brevity):

storage.objects.get
storage.objects.list
…

The structure of permissions is usually <service>.<resource type>.<action>.

GCP has a large number of predefined roles, but users can also create custom roles inside their GCP projects.

For an identity to be allowed to perform an action on a resource, the identity needs to be granted, on that resource, a role that contains the permission required for that action. The role needs to be granted directly on the resource, or on a parent resource in the resource hierarchy:

Resource
Project (set of resources)
Folder (set of projects)
Organization (set of folders)

If the role is granted on a parent resource, the relationship will be inherited by all the descendants. For example, if a role is granted on a folder, all the projects and resources inside that folder will inherit the grant.

Reminder: The grant relationship has three involved entities:

The identity that will use the permissions to perform actions.

The resource that will be the target of the actions.

The role containing the permissions required for the actions.

Granting permissions

When granting roles, GCP IAM uses the term Principal, formerly Member, to refer to the string that globally identifies the identity which will receive the grant.

To GCP Service Accounts

The IAM Principal for a GCP Service Account is the email address of the Service Account prefixed with serviceAccount::

serviceAccount:SA_NAME@PROJECT_ID.iam.gserviceaccount.com

To Kubernetes Service Accounts

The IAM Principal for a Kubernetes Service Account from a GKE cluster has the following format:

serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]

Flux running inside non-GKE Kubernetes clusters (e.g. EKS, AKS, kind running locally or in CI, etc.) can also get access to GCP resources. For Service Accounts from such clusters, the IAM Principal has the following format:

principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/system:serviceaccount:NAMESPACE:KSA_NAME

Where the portion projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID globally identifies the Workload Identity Pool configured for the non-GKE cluster.

See more details about how this works in the GCP docs Workload Identity Federation with Kubernetes and how to configure this for Flux in the Workload Identity Federation section.

Via Config Connector IAM custom resources

To use Config Connector for granting roles you can use one of the following custom resources:

All of them support both the Organization, Folder and Project high-level hierarchical resource kinds, and the kinds for all the individual resource types that Flux integrates with. For the individual resource types, see the Authorization section below.

Note: Config Connector is the most GitOps-friendly way of managing GCP resources. With it you can even use Flux itself to deploy and continuously reconcile your GCP custom resources (the Kubernetes ones), and Config Connector itself will continuously reconcile the actual GCP resources.

Via Terraform/OpenTofu IAM resources

To use Terraform/OpenTofu for granting roles you can use one of the following resources:

google_<resource type>_iam_policy
google_<resource type>_iam_binding
google_<resource type>_iam_member

See the docs for the high-level hierarchical resources:

For the Terraform/OpenTofu resources for the individual resource types that Flux integrates with, see the Authorization section below.

Note: Terraform/OpenTofu is a great declarative way of managing GCP resources, but it’s not as GitOps-friendly as Config Connector, as these tools do not prescribe continuous reconciliation of the resources. There are alternatives to achieve that, but they are not a core part of the tools.

Via the `gcloud` CLI

To use the gcloud CLI for granting roles you can use one of the following commands:

gcloud <resource type command> set-iam-policy
gcloud <resource type command> add-iam-policy-binding

See the docs for the high-level hierarchical resource commands:

For the gcloud commands for the individual resource types that Flux integrates with, see the Authorization section below.

Note: The gcloud CLI is a great tool for experimenting with GCP resources and their integrations with Flux, but it’s not a GitOps-friendly way of managing GCP resources. If you need continuous reconciliation of the resources, prefer using Config Connector or Terraform/OpenTofu.

Authorization

In this section we describe the recommended roles and permissions for enabling the Flux integrations with GCP services.

For Google Cloud Artifact Registry

The OCIRepository, ImageRepository and ImagePolicy Flux APIs are integrated with GAR. The OCIRepository API can be used to pull OCI artifacts from GAR repositories into the cluster, while the ImageRepository and ImagePolicy APIs can be used to reflect tags and digests of such artifacts also inside the cluster.

The recommended role containing the required permissions for the OCIRepository, ImageRepository and ImagePolicy APIs is:

Artifact Registry Reader (roles/artifactregistry.reader)

The Config Connector resource kinds for use with the IAM custom resources are:

ArtifactRegistryRepository

The Terraform/OpenTofu IAM resource types are:

artifact_registry_repository

The commands for the gcloud CLI are:

artifacts repositories

For Google Cloud Storage

The Bucket Flux API is integrated with GCS. The Bucket API can be used to pull manifests from GCS buckets and package them as artifacts inside the cluster.

The recommended roles containing the required permissions for the Bucket API are:

In the specific case of the Bucket API, both roles are needed because Flux needs to confirm the existence of the bucket. If the bucket does not exist Flux can error out earlier in the reconciliation process and give an error message that is easier to debug.

Alternatively, you can create a custom role containing the following permissions:

storage.buckets.get
storage.objects.get
storage.objects.list

The Config Connector resource kinds for use with the IAM custom resources are:

StorageBucket

The Terraform/OpenTofu IAM resource types are:

storage_bucket

The commands for the gcloud CLI are:

storage buckets

For Google Cloud Key Management Service

The Kustomization Flux API is integrated with KMS. The Kustomization API is used to apply manifests in the cluster, and it can use KMS to decrypt SOPS-encrypted secrets before applying them.

The recommended role containing the required permissions for the Kustomization API is:

Cloud KMS CryptoKey Decrypter (roles/cloudkms.cryptoKeyDecrypter)

This role can be granted either on a Key Ring (a set of keys), or on an individual Crypto Key.

The Config Connector resource kinds for use with the IAM custom resources are:

The Terraform/OpenTofu IAM resource types are:

The commands for the gcloud CLI are:

For Google Kubernetes Engine

The Kustomization and HelmRelease Flux APIs can use IAM Principals for applying and managing resources in remote GKE clusters. Two kinds of access must be configured for this:

The IAM Principal must have permission to call the GetCluster GKE API for the target remote cluster. The Flux controllers need to call this API for retrieving details required for connecting to the remote cluster, like the cluster’s API server endpoint and certificate authority data. This is done by granting an IAM role to the IAM Principal that allows this action on the target remote cluster.
The IAM Principal must have permissions inside the remote cluster to apply and manage the target resources. There are two ways of granting these permissions: via Kubernetes RBAC, or via IAM roles. The former means simply referencing the IAM Principal in RoleBinding or ClusterRoleBinding objects inside the remote cluster as the Kubernetes username. The latter means granting IAM roles that grant Kubernetes permissions to the IAM Principal on the target remote cluster. The resulting set of permissions granted to the IAM Principal will be the union of the permissions granted via Kubernetes RBAC with the permissions granted via IAM roles.

In GCP a GKE cluster is regarded as a project-level resource, so IAM roles granting access to GKE APIs can only be granted at the project level or higher in the resource hierarchy.

Permissions for the GKE API

The recommended role containing the required permissions for calling the GetCluster GKE API is:

Kubernetes Engine Cluster Viewer (roles/container.clusterViewer)

Alternatively, you can create a custom role containing the following permission:

container.clusters.get

Permissions inside the remote cluster

For granting Kubernetes RBAC to the IAM Principal, simply create the corresponding RoleBinding or ClusterRoleBinding objects inside the remote cluster using the principal as the Kubernetes username.

For granting cluster-scoped permissions through IAM roles, you can grant either built-in roles or custom roles.

For an exhaustive list of the permissions that can be added to a custom IAM role for cluster-scoped permissions, go to the page of the Kubernetes Engine Admin role and expand the collapsed section container.* at the very top of the Permissions column.

Note: Namespaced permissions can only be granted via Kubernetes RBAC. IAM roles can only grant cluster-scoped permissions. See docs.

For Google Cloud Pub/Sub

The Provider Flux API is integrated with Pub/Sub. The Provider API can be used to send notifications about Flux resources to Pub/Sub topics.

The recommended role containing the required permissions for the Provider API is:

Pub/Sub Publisher (roles/pubsub.publisher)

The Config Connector resource kinds for use with the IAM custom resources are:

PubSubTopic

The Terraform/OpenTofu IAM resource types are:

pubsub_topic

The commands for the gcloud CLI are:

pubsub topics

Authentication

As mentioned in the Identity section, GCP supports two types of identities for Flux: GCP Service Accounts and Kubernetes Service Accounts. This section describes how to authenticate each type of identity. Both types support Workload Identity Federation (secret-less), but only GCP Service Accounts support secret-based authentication.

Recommendation: Always prefer secret-less over secret-based authentication if the alternative is available. Secrets can be stolen to abuse the permissions granted to the identities they represent, and for public clouds like GCP this can be done by simply having Internet access. This requires secrets to be regularly rotated and more security controls to be put in place, like audit logs, secret management tools, etc. Secret-less authentication does not have this problem, as the identity is authenticated using a token that is not stored anywhere and is only valid for a short period of time, usually one hour. It’s much harder to steal an identity this way.

With Workload Identity Federation

Workload Identity Federation is a GCP feature that allows external identities to authenticate with GCP services without the need for a per-identity static credential. This is done by exchanging a short-lived token issued by the external identity provider (in this case, Kubernetes) for a short-lived Google OAuth 2.0 Access Token. This access token is then used to authenticate with GCP services. This process is more broadly known as OIDC federation.

Supported clusters

GCP supports Workload Identity Federation for both GKE and non-GKE clusters.

In GKE Autopilot, Workload Identity Federation is always enabled, no cluster setup is required. In GKE Standard, follow these docs.

For non-GKE clusters you need to create a Workload Identity Pool and a Workload Identity Provider with the Issuer URL of the cluster.

To create a Workload Identity Pool and a Workload Identity Provider using Config Connector, you can use the following custom resources:

To create a Workload Identity Pool and a Workload Identity Provider using Terraform/OpenTofu, you can use the following resources:

To create a Workload Identity Pool and a Workload Identity Provider using the gcloud CLI, you can use the following commands:

Supported identity types

When using Workload Identity Federation, Kubernetes Service Accounts are the identity type used by default. In this case, roles must be granted using the IAM Principal formats described in this section, with the appropriate format depending on whether the cluster is GKE or not.

You can also optionally use the roles granted to a GCP Service Account instead of the roles granted to a Kubernetes Service Account with Workload Identity Federation, but a Kubernetes Service Account is still required. This is done by a process called impersonation. To configure a Kubernetes Service Account to impersonate a GCP Service Account, two steps are required:

Allow the Kubernetes Service Account to impersonate the GCP Service Account. This is done by granting the Workload Identity User (roles/iam.workloadIdentityUser) role to the Kubernetes Service Account on the GCP Service Account. The IAM Principals required for this are the same described here.

To perform this step using Config Connector, you can use one of the IAM custom resources mentioned here with the following kind:

IAMServiceAccount

To perform this step using Terraform/OpenTofu, you can use one of the IAM resources mentioned here with the following resource type:

service_account

To perform this step using the gcloud CLI, you can use one of the IAM commands mentioned here with the following command:

iam service-accounts

Annotate the Kubernetes Service Account with the GCP Service Account email. This is done by adding the annotation iam.gke.io/gcp-service-account: SA_NAME@PROJECT_ID.iam.gserviceaccount.com to the Kubernetes Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: KSA_NAME
 namespace: NAMESPACE
 annotations:
 iam.gke.io/gcp-service-account: SA_NAME@PROJECT_ID.iam.gserviceaccount.com

Configuring Flux to use a Kubernetes Service Account to authenticate with GCP, regardless if it’s configured to impersonate a GCP Service Account or not, can be done either at the object level or at the controller level.

With GCP Service Account Keys

All GCP integrations except for GAR support configuring authentication through a GCP Service Account Key.

GCP Service Accounts support static private keys that can be used to generate temporary access tokens for authenticating with GCP services. Such a key is exported as a JSON document containing the key itself and other metadata.

The best way to create a GCP Service Account Key is using the following Config Connector custom resource:

IAMServiceAccountKey

This custom resource will automatically create a Kubernetes Secret inside the cluster containing the GCP Service Account Key in the JSON format.

To create the key using Terraform/OpenTofu, you can use the following resource:

google_service_account_key

To create the key using the gcloud CLI, you can use the following command:

iam service-accounts keys create

Configuring Flux to use a GCP Service Account Key can be done either at the object level or at the controller level.

At the object level

For Workload Identity Federation

Before following the steps below, make sure to complete the cluster setup described here, and to configure the Kubernetes Service Account as described here.

For configuring authentication through a Kubernetes Service Account at the object level, regardless if the Kubernetes Service Account is configured to impersonate a GCP Service Account or not, the following steps are required:

Enable the feature gate ObjectLevelWorkloadIdentity in the target Flux controller Deployment during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: (some-controller)
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --feature-gates=ObjectLevelWorkloadIdentity=true

Set the .spec.provider field to gcp in the Flux resource. For SOPS decryption with the Kustomization API, this field is not required/does not exist, SOPS detects the provider from the metadata in the SOPS-encrypted Secret. For remote cluster access with the Kustomization and HelmRelease APIs the field is .data.provider inside the referenced ConfigMap from .spec.kubeConfig.configMapRef.
Use the .spec.serviceAccountName field to specify the name of the Kubernetes Service Account in the same namespace as the Flux resource. For SOPS decryption with the Kustomization API, the field is .spec.decryption.serviceAccountName. For remote cluster access with the Kustomization and HelmRelease APIs the field is .data.serviceAccountName inside the referenced ConfigMap from .spec.kubeConfig.configMapRef.
Only if the cluster is not GKE, annotate the Kubernetes Service Account with the fully qualified name of the Workload Identity Provider:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: KSA_NAME
 namespace: NAMESPACE
 annotations:
 iam.gke.io/gcp-service-account: SA_NAME@PROJECT_ID.iam.gserviceaccount.com # GCP Service Account impersonation is optional
 gcp.auth.fluxcd.io/workload-identity-provider: projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID

Note: The iam.gke.io/gcp-service-account annotation is defined by GKE, but Flux also uses it to identify the GCP Service Account to impersonate in non-GKE clusters. This is for providing users with a seamless experience.

For GCP Service Account Keys

All GCP integrations except for GAR and GKE support configuring authentication through a GCP Service Account Key at the object level.

For configuring authentication through a GCP Service Account Key, the .spec.secretRef.name field must be set to the name of the Kubernetes Secret in the same namespace as the Flux resource containing the GCP Service Account Key. In the case of the Kustomization API, the field is .spec.decryption.secretRef.name.

The provider field and the key inside the .data field of the Secret depend on the specific Flux API:

For the Bucket API the .spec.provider field must be set to gcp, and the key inside the .data field of the Secret must be serviceaccount.
For the Provider API, the .spec.type field must be set to googlepubsub and the key inside the .data field of the Secret must be token.
For SOPS decryption with the Kustomization API, there’s no provider field as SOPS detects the provider from the metadata in the SOPS-encrypted Secret. The key inside the .data field of the Secret must be sops.gcp-kms.

At the controller level

At the controller level, regardless if authenticating with Workload Identity Federation or GCP Service Account Keys, all Flux resources must have the provider field set to gcp according to the rules below.

For all Flux resource kinds except for Kustomization and HelmRelease, set the .spec.provider field to gcp and leave .spec.serviceAccountName unset.

For SOPS decryption with the Kustomization API, leave the .spec.decryption.serviceAccountName field unset. There’s no provider field for SOPS decryption.

For remote cluster access with the Kustomization and HelmRelease APIs, set the .data.provider field of the ConfigMap referenced by the .spec.kubeConfig.configMapRef field to gcp and leave the .data.serviceAccountName field unset.

The controller-level configuration is described in the following sections.

For Workload Identity Federation

Before following the steps below, make sure to complete the cluster setup described here, and to configure the Kubernetes Service Account as described here.

If the cluster is GKE, the Kubernetes Service Account of the controller can optionally be configured to impersonate a GCP Service Account. This is done by adding the iam.gke.io/gcp-service-account annotation to the controller Service Account during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: ServiceAccount
 name: "(some-controller)"
 patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 iam.gke.io/gcp-service-account: SA_NAME@PROJECT_ID.iam.gserviceaccount.com # GCP Service Account impersonation is optional

If the configuration above is done after bootstrap, restart (delete) the controller for the binding to take effect.

Hint: As discussed in the Workload Identity Federation section, impersonating a GCP Service Account is entirely optional. If not configured, the controller Kubernetes Service Account will be used for authentication, in which case the roles must be granted directly to it according to what is described in this section.

If the cluster is not GKE, the controller Deployment must be patched during bootstrap according to these docs:

A projected volume must be mounted in the controller Deployment with a Kubernetes Service Account token whose audience is set to the Workload Identity Provider URL.
A ConfigMap volume must be mounted in the controller Deployment with the JSON configuration below.
The environment variable GOOGLE_APPLICATION_CREDENTIALS must be set to the path of the mounted JSON configuration file.

The controller patch should look like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: "(some-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: GOOGLE_APPLICATION_CREDENTIALS
 value: /etc/workload-identity/credential-configuration.json
 - op: add
 path: /spec/template/spec/volumes/-
 value:
 name: gcp-token
 projected:
 sources:
 - serviceAccountToken:
 audience: https://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID
 expirationSeconds: 3600
 path: gcp-token
 - op: add
 path: /spec/template/spec/volumes/-
 value:
 name: workload-identity-credential-configuration
 configMap:
 name: some-controller-workload-identity-federation
 - op: add
 path: /spec/template/spec/containers/0/volumeMounts/-
 value:
 name: gcp-token
 mountPath: /var/run/service-account
 readOnly: true
 - op: add
 path: /spec/template/spec/containers/0/volumeMounts/-
 value:
 name: workload-identity-credential-configuration
 mountPath: /etc/workload-identity
 readOnly: true

For direct access using the roles granted to the controller Kubernetes Service account, the ConfigMap containing the JSON configuration should look like this (it should contain the token_info_url field set to https://sts.googleapis.com/v1/introspect):

apiVersion: v1
kind: ConfigMap
metadata:
 name: some-controller-workload-identity-federation
 namespace: flux-system
data:
 credential-configuration.json: |
 {
 "universe_domain": "googleapis.com",
 "type": "external_account",
 "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
 "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
 "token_url": "https://sts.googleapis.com/v1/token",
 "token_info_url": "https://sts.googleapis.com/v1/introspect",
 "credential_source": {
 "file": "/var/run/service-account/gcp-token",
 "format": {
 "type": "text"
 }
 }
 }

For impersonating a GCP Service Account using the controller Kubernetes Service Account, the ConfigMap containing the JSON configuration should look like this (it should contain the service_account_impersonation_url field instead of the token_info_url field):

apiVersion: v1
kind: ConfigMap
metadata:
 name: some-controller-workload-identity-federation
 namespace: flux-system
data:
 credential-configuration.json: |
 {
 "universe_domain": "googleapis.com",
 "type": "external_account",
 "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
 "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
 "token_url": "https://sts.googleapis.com/v1/token",
 "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SA_NAME@PROJECT_ID.iam.gserviceaccount.com:generateAccessToken",
 "credential_source": {
 "file": "/var/run/service-account/gcp-token",
 "format": {
 "type": "text"
 }
 }
 }

For GCP Service Account Keys

Mount the Kubernetes Secret containing the GCP Service Account Key inside the controller Deployment as a volume, and set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the mounted JSON file during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: "(some-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: GOOGLE_APPLICATION_CREDENTIALS
 value: /etc/gcp-service-account/key.json
 - op: add
 path: /spec/template/spec/volumes/-
 value:
 name: gcp-service-account
 secret:
 secretName: some-controller-gcp-service-account
 - op: add
 path: /spec/template/spec/containers/0/volumeMounts/-
 value:
 name: gcp-service-account
 mountPath: /etc/gcp-service-account
 readOnly: true

The Secret should look like this:

apiVersion: v1
kind: Secret
metadata:
 name: some-controller-gcp-service-account
 namespace: flux-system
type: Opaque
stringData:
 key.json: |
 JSON_GCP_SERVICE_ACCOUNT_KEY

At the node level

Only for the GAR integrations Flux supports authentication at the node level for GKE. This is because users often already have to configure authentication at the node level for GKE to be able to pull container images from GAR in order to start pods. By supporting this authentication method Flux allows users to configure GAR authentication in a single way. See docs.

⚠️ Node level authentication may work for other integrations as well, but Flux only has continuous integration tests for the GAR integration in order to support the specific use case described above.

For node-level authentication to work, the .spec.provider field of the Flux resources must be set to gcp.

Flux: Microsoft Azure

Mon, 01 Jan 0001 00:00:00 +0000

The Flux APIs integrate with the following Microsoft Azure services:

The source-controller integrates the OCIRepository API with Azure Container Registry (ACR) for pulling OCI artifacts into the cluster.
The image-reflector-controller integrates the ImageRepository and ImagePolicy APIs with ACR for scanning tags and digests of OCI artifacts and reflecting them into the cluster.
The source-controller integrates the GitRepository API with Azure DevOps (ADO) for pulling manifests from Git repositories and packaging them as artifacts inside the cluster.
The image-automation-controller integrates the ImageUpdateAutomation API with ADO for automating image updates in Git repositories.
The notification-controller integrates the Provider API with ADO for updating commit statuses in Git repositories upon events related to Flux resources.
The source-controller integrates the Bucket API with Azure Blob Storage (ABS) for pulling manifests from containers and packaging them as artifacts inside the cluster.
The kustomize-controller integrates the Kustomization API with Azure Key Vault (AKV) for decrypting SOPS-encrypted secrets when applying manifests in the cluster.
The kustomize-controller integrates the Kustomization API with Azure Kubernetes Service (AKS) for applying manifests in remote AKS clusters.
The helm-controller integrates the HelmRelease API with AKS for applying Helm charts in remote AKS clusters.
The notification-controller integrates the Provider API with Azure Event Hubs (AEH) for sending notifications about Flux resources outside the cluster.

The next sections briefly describe Microsoft Entra ID, formerly known as Azure Active Directory (AAD), and Azure RBAC, Azure’s identity and access management systems, and how they can be used to grant Flux access to resources offered by the services above. Bear in mind that Microsoft Entra ID and Azure RBAC have more features than the ones described here. We describe only the features that are relevant for the Flux integrations.

Note: Flux also supports Microsoft Teams webhooks, but Microsoft Teams webhooks do not support Microsoft Entra ID. For more information about the notification-controller integration with Microsoft Teams, see the Provider API docs.

Identity

For all the integrations with Azure, Microsoft Entra ID will authenticate two types of identities for Flux:

For both types a Service Principal can be enabled, with each identity having its own Service Principal. The Service Principal is the entity of an identity that is used to integrate with other Azure systems, such as Azure RBAC and ADO.

Also, each identity type has its own methods of authentication. In particular, Managed Identities support secret-less authentication, while Applications support secret-based authentication. The former is more secure.

For further understanding how to configure authentication for both types of identities, refer to the Authentication section.

Access Management

Azure Role-Based Access Control (RBAC) is the system that Azure employs to manage access for most of its resources. Some services, such as ADO, do not use Azure RBAC and instead have their own access control system.

Identities in Azure need permissions to access resources from Azure services. Permissions cannot be granted directly to identities. Instead, Azure RBAC uses roles to group permissions, and only roles can be granted to identities. Each role has a display name and a globally unique ID. For example, the role Storage Blob Data Reader, whose ID is /providers/Microsoft.Authorization/roleDefinitions/2a2b9908-6ea1-4ae2-8e65-a410df84e7d1, has the following definition:

{
 "assignableScopes": [
 "/"
 ],
 "description": "Allows for read access to Azure Storage blob containers and data",
 "id": "/providers/Microsoft.Authorization/roleDefinitions/2a2b9908-6ea1-4ae2-8e65-a410df84e7d1",
 "name": "2a2b9908-6ea1-4ae2-8e65-a410df84e7d1",
 "permissions": [
 {
 "actions": [
 "Microsoft.Storage/storageAccounts/blobServices/containers/read",
 "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action"
 ],
 "notActions": [],
 "dataActions": [
 "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read"
 ],
 "notDataActions": []
 }
 ],
 "roleName": "Storage Blob Data Reader",
 "roleType": "BuiltInRole",
 "type": "Microsoft.Authorization/roleDefinitions"
}

The structure of permissions is usually Microsoft.<service>/<resource type>/<action> and they are grouped into two categories:

Data actions: These actions are related to data operations, such as reading or writing data.
Non-data actions: These actions are related to management operations, such as creating or deleting resources.

Azure has a large number of predefined roles, but users can also create custom roles inside their Azure subscriptions.

For an identity to be allowed to perform an action on a resource, the identity needs to be granted, on a scope covering that resource, a role that contains the permission required for that action. The role needs to be granted directly on the resource scope, or on the scope of a parent resource in the resource hierarchy:

Resource
Resource Group (set of resources)
Subscription (set of resource groups)
Management Group (set of subscriptions)

If the role is granted on a parent resource, the relationship will be inherited by all the descendants. For example, if a role is granted on a subscription, all the resource groups and resources inside that subscription will inherit the grant.

Reminder: The grant relationship has three involved entities:

The identity that will use the permissions to perform actions.

The scope covering the resource that will be the target of the actions.

The role containing the permissions required for the actions.

Granting permissions

Azure RBAC grants roles to Microsoft Entra identities through the Service Principal of the identity. The principal ID of an identity can be looked up through the client ID (also known as application ID) of the identity.

To find a principal ID using Terraform/OpenTofu, use the client ID in the following data source (look for the object_id attribute):

azuread_service_principal

To find a principal ID using the az CLI, use the client ID in the following command:

az ad sp show --id <client-id> --query id -o tsv

Via Azure Service Operator custom resources

To use Azure Service Operator (ASO) for granting roles you can use the following custom resource:

RoleAssignment

Note: ASO is the most GitOps-friendly way of managing Azure resources. With it you can even use Flux itself to deploy and continuously reconcile your Azure custom resources (the Kubernetes ones), and ASO itself will continuously reconcile the actual Azure resources.

Via Terraform/OpenTofu resources

To use Terraform/OpenTofu for granting roles you can use the following resource:

azurerm_role_assignment

Note: Terraform/OpenTofu is a great declarative way of managing Azure resources, but it’s not as GitOps-friendly as ASO, as these tools do not prescribe continuous reconciliation of the resources. There are alternatives to achieve that, but they are not a core part of the tools.

Via the `az` CLI

To use the az CLI for granting roles you can use the following command:

az role assignment create

Note: The az CLI is a great tool for experimenting with Azure resources and their integrations with Flux, but it’s not a GitOps-friendly way of managing Azure resources. If you need continuous reconciliation of the resources, prefer using ASO or Terraform/OpenTofu.

Authorization

In this section we describe the recommended roles and permissions for enabling the Flux integrations with Azure services.

For Azure Container Registry

The OCIRepository, ImageRepository and ImagePolicy Flux APIs are integrated with ACR. The OCIRepository API can be used to pull OCI artifacts from ACR repositories into the cluster, while the ImageRepository and ImagePolicy APIs can be used to reflect tags and digests of such artifacts also inside the cluster.

The recommended role containing the required permissions for the OCIRepository, ImageRepository and ImagePolicy APIs is:

AcrPull (/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d)

The scopes on which the AcrPull role can be granted are:

Registry: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.ContainerRegistry/registries/REGISTRY_NAME
Resource Group: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME
Subscription: /subscriptions/SUBSCRIPTION_ID
Management Group: /providers/Microsoft.Management/managementGroups/MANAGEMENT_GROUP_NAME

For Azure DevOps

The GitRepository, ImageUpdateAutomation and Provider Flux APIs are integrated with ADO. The GitRepository API can be used to pull manifests from ADO Git repositories and package them as artifacts inside the cluster, the ImageUpdateAutomation API can be used to automate image updates in ADO Git repositories, and the Provider API can be used to update commit statuses in ADO Git repositories upon events related to Flux resources.

While ADO integrates with Microsoft Entra ID Service Principals, it does not integrate with Azure RBAC. As mentioned before, ADO has its own access control system.

The Service Principal must be onboarded into the ADO organization and added to a group. For the GitRepository API, the Readers group is sufficient, while for the ImageUpdateAutomation and Provider APIs the Contributors group is required for pushing commits to the repository and updating commit statuses, respectively.

ASO does not support managing ADO resources (see issue).

Via Terraform/OpenTofu resources

To grant access to Service Principals using Terraform/OpenTofu, first you need to configure the ADO provider with the ADO organization and project:

azuredevops

Then you can use the following resources:

azuredevops_service_principal_entitlement: This resource onboards a Service Principal into an ADO organization.
azuredevops_group_membership: This resource adds the Service Principal to an ADO group.

Use the descriptor attribute of the azuredevops_service_principal_entitlement in the members field of the azuredevops_group_membership resource.

Use the azuredevops_group data source to find the ID of the desired group, and use it in the group field of the azuredevops_group_membership resource.

Via the `az` CLI

To grant access to Service Principals using the az CLI, you can use the following commands:

Login to ADO: az devops login
Configure the ADO organization and project: az devops configure
Onboard the Service Principal into the ADO organization. This doesn’t have a dedicated command yet, so use az rest to call the REST API for Adding Service Principal Entitlements.
Find the member ID of the Service Principal in the organization using az rest with the REST API for Querying Subjects, the query field of the request body should be set to the principal ID of the Service Principal. The response should be a list of GraphSubject, the member ID will be the descriptor field.
List the groups in the organization to find the ID of the desired group: az devops security group list
Add the Service Principal to the group: az devops security group membership add

For Azure Blob Storage

The Bucket Flux API is integrated with ABS. The Bucket API can be used to pull manifests from ABS containers and package them as artifacts inside the cluster.

The recommended role containing the required permissions for the Bucket API is:

Storage Blob Data Reader (/providers/Microsoft.Authorization/roleDefinitions/2a2b9908-6ea1-4ae2-8e65-a410df84e7d1)

The scopes on which the Storage Blob Data Reader role can be granted are:

Container: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.Storage/storageAccounts/STORAGE_ACCOUNT_NAME/blobServices/default/containers/CONTAINER_NAME
Storage Account: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.Storage/storageAccounts/STORAGE_ACCOUNT_NAME
Resource Group: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME
Subscription: /subscriptions/SUBSCRIPTION_ID
Management Group: /providers/Microsoft.Management/managementGroups/MANAGEMENT_GROUP_NAME

For Azure Key Vault

The Kustomization Flux API is integrated with AKV. The Kustomization API is used to apply manifests in the cluster, and it can use AKV to decrypt SOPS-encrypted secrets before applying them.

The recommended role containing the required permissions for the Kustomization API is:

Key Vault Crypto User (/providers/Microsoft.Authorization/roleDefinitions/12338af0-0e69-4776-bea7-57ae8d297424)

The scopes on which the Key Vault Crypto User role can be granted are:

Vault: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.KeyVault/vaults/VAULT_NAME
Resource Group: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME
Subscription: /subscriptions/SUBSCRIPTION_ID
Management Group: /providers/Microsoft.Management/managementGroups/MANAGEMENT_GROUP_NAME

For Azure Kubernetes Service

The Kustomization and HelmRelease Flux APIs can use Microsoft Entra identities for applying and managing resources in remote AKS clusters. The AKS cluster must be integrated with Microsoft Entra ID for this feature work.

Note: If you have a reason not to enable the Microsoft Entra ID integration in your cluster, you can still fetch the static cluster-admin kubeconfig from Azure and use it with the .spec.kubeConfig.secretRef field of the Kustomization and HelmRelease APIs.

Two kinds of access must be configured for the Microsoft Entra identity:

The Microsoft Entra identity must have permission to call the Get and ListClusterUserCredentials AKS APIs for the target remote cluster. The Flux controllers need to call these APIs for retrieving details required for connecting to the remote cluster, like the cluster’s API server endpoint and certificate authority data. This is done by granting an Azure RBAC role to the Microsoft Entra identity that allows these actions on the target remote cluster.
The Microsoft Entra identity must have permissions inside the remote cluster to apply and manage the target resources. There are two ways of granting these permissions: via Kubernetes RBAC, or via Azure RBAC. The former means simply referencing the principal ID of the Microsoft Entra identity in RoleBinding or ClusterRoleBinding objects inside the remote cluster as the Kubernetes username. The latter means granting Azure RBAC roles that grant Kubernetes permissions to the Microsoft Entra identity on the target remote cluster. The resulting set of permissions granted to the Microsoft Entra identity will be the union of the permissions granted via Kubernetes RBAC with the permissions granted via Azure RBAC.

Permissions for the AKS APIs

The recommended role containing the required permissions for calling the Get and ListClusterUserCredentials AKS APIs is:

Azure Kubernetes Service Cluster User Role (/providers/Microsoft.Authorization/roleDefinitions/4abbcc35-e782-43d8-92c5-2d3f1bd2253f)

The scopes on which the Azure Kubernetes Service Cluster User Role role can be granted are:

Cluster: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.ContainerService/managedClusters/CLUSTER_NAME
Resource Group: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME
Subscription: /subscriptions/SUBSCRIPTION_ID
Management Group: /providers/Microsoft.Management/managementGroups/MANAGEMENT_GROUP_NAME

Permissions inside the remote cluster

For granting Kubernetes RBAC to the principal ID of a Microsoft Entra identity, simply create the corresponding RoleBinding or ClusterRoleBinding objects inside the remote cluster using the principal ID as the Kubernetes username.

For granting permissions through Azure RBAC roles, you can grant either built-in roles or custom roles to Microsoft Entra identity on the following scopes:

Namespace: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.ContainerService/managedClusters/CLUSTER_NAME/namespaces/NAMESPACE_NAME
Cluster: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.ContainerService/managedClusters/CLUSTER_NAME
Resource Group: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME
Subscription: /subscriptions/SUBSCRIPTION_ID
Management Group: /providers/Microsoft.Management/managementGroups/MANAGEMENT_GROUP_NAME

Azure RBAC must be enabled for the remote AKS cluster.

For Azure Event Hubs

The Provider Flux API is integrated with AEH. The Provider API can be used to send notifications about Flux resources to Event Hubs.

The recommended role containing the required permissions for the Provider API is:

Azure Event Hubs Data Sender (/providers/Microsoft.Authorization/roleDefinitions/2b629674-e913-4c01-ae53-ef4638d8f975)

The scopes on which the Azure Event Hubs Data Sender role can be granted are:

Event Hub: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.EventHub/namespaces/NAMESPACE_NAME/eventhubs/EVENTHUB_NAME
Namespace: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME/providers/Microsoft.EventHub/namespaces/NAMESPACE_NAME
Resource Group: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP_NAME
Subscription: /subscriptions/SUBSCRIPTION_ID
Management Group: /providers/Microsoft.Management/managementGroups/MANAGEMENT_GROUP_NAME

Authentication

As mentioned in the Identity section, Azure supports two types of identities for Flux: Managed Identities and Applications. This section describes how to authenticate each type of identity. Managed Identities are the recommended way of authenticating to Azure services, as they support secret-less authentication. Applications are not recommended, as they require secret-based authentication, which is less secure.

Recommendation: Always prefer secret-less over secret-based authentication if the alternative is available. Secrets can be stolen to abuse the permissions granted to the identities they represent, and for public clouds like Azure this can be done by simply having Internet access. This requires secrets to be regularly rotated and more security controls to be put in place, like audit logs, secret management tools, etc. Secret-less authentication does not have this problem, as the identity is authenticated using a token that is not stored anywhere and is only valid for a short period of time, usually one hour. It’s much harder to steal an identity this way.

With Workload Identity Federation

Workload Identity Federation is an Azure feature that allows external identities to authenticate with Azure services without the need for a per-identity static credential. This is done by exchanging a short-lived token issued by the external identity provider (in this case, Kubernetes) for a short-lived Azure Access Token. This access token is then used to authenticate with Azure services. This process is more broadly known as OIDC federation and is supported only for Managed Identities.

Supported clusters

Azure supports Workload Identity Federation for both AKS and non-AKS clusters.

In AKS clusters you need to enable the options OIDC Issuer and Workload Identity.

For both AKS and non-AKS clusters you need to know the Issuer URL of the cluster, which is an input required for creating Federated Credentials (see below).

Supported identity types

As mentioned before, the only identity type supported by Workload Identity Federation is Managed Identities.

Flux acquires the roles granted to a Managed Identity by using a Kubernetes Service Account to impersonate this Managed Identity. To configure a Kubernetes Service Account to impersonate a Managed Identity, two steps are required:

Allow the Kubernetes Service Account to impersonate the Managed Identity. This is done by creating a Federated Credential that ties together the Kubernetes Service Account and the Managed Identity.

The inputs identifying the Managed Identity for creating the Federated Credential are:

The Resource Group.
The Name of the Managed Identity inside the Resource Group.

The inputs identifying the Kubernetes Service Account for creating the Federated Credential are:

The Issuer URL of the cluster.
The Subject string that Kubernetes uses to identify the Service Account, which is system:serviceaccount:NAMESPACE:KSA_NAME.

Additionally, the Federated Credential needs:

A name for itself.
The Audience to be set to api://AzureADTokenExchange.

To create a Federated Credential using ASO, you can use the following custom resource:

FederatedIdentityCredential

To create a Federated Credential using Terraform/OpenTofu, you can use the following resource:

azurerm_federated_identity_credential

to create a Federated Credential using the az CLI, you can use the following command:

az identity federated-credential create

Annotate the Kubernetes Service Account with the Managed Identity Client ID and Tenant ID. This is done by adding the annotations azure.workload.identity/client-id: CLIENT_ID and azure.workload.identity/tenant-id: TENANT_ID to the Kubernetes Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: KSA_NAME
 namespace: NAMESPACE
 annotations:
 azure.workload.identity/client-id: CLIENT_ID
 azure.workload.identity/tenant-id: TENANT_ID

Configuring Flux to use a Kubernetes Service Account to authenticate with Azure can be done either at the object level or at the controller level.

With Application Certificates

All Azure integrations except for ACR support configuring authentication through an Application Certificate.

Applications support static certificates that can be used to generate temporary access tokens for authenticating with Azure services.

ASO does not support creating Application Certificates (see issue).

To create an Application Certificate using Terraform/OpenTofu, you can use the following resource:

azuread_service_principal_certificate

To create or reset an Application Certificate using the az CLI, you can use the following command:

az ad sp credential reset --create-cert

Configuring Flux to use an Application Certificate can be done either at the object level or at the controller level.

At the object level

For Workload Identity Federation

Before following the steps below, make sure to complete the cluster setup described here, and to configure the Kubernetes Service Account as described here.

⚠️ Attention: Only for AKS clusters targeting Azure resources object-level workload identity is currently unstable. In its current state the feature may stop working without notice depending on actions taken by Azure to meet AKS customers’ demands. For continued support the feature may need to evolve in a way that additional configuration steps will be required for it to continue working in future Flux releases. See this issue for more information.

For configuring authentication through a Kubernetes Service Account at the object level the following steps are required:

Enable the feature gate ObjectLevelWorkloadIdentity in the target Flux controller Deployment during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: (some-controller)
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --feature-gates=ObjectLevelWorkloadIdentity=true

Set the .spec.provider field to azure in the Flux resource. For SOPS decryption with the Kustomization API, this field is not required/does not exist, SOPS detects the provider from the metadata in the SOPS-encrypted Secret. For remote cluster access with the Kustomization and HelmRelease APIs the field is .data.provider inside the referenced ConfigMap from .spec.kubeConfig.configMapRef.
Use the .spec.serviceAccountName field to specify the name of the Kubernetes Service Account in the same namespace as the Flux resource. For SOPS decryption with the Kustomization API, the field is .spec.decryption.serviceAccountName. For remote cluster access with the Kustomization and HelmRelease APIs the field is .data.serviceAccountName inside the referenced ConfigMap from .spec.kubeConfig.configMapRef.

Note: The azure.workload.identity/client-id and azure.workload.identity/tenant-id annotations are defined by AKS, but Flux also uses them to identify the Managed Identity to impersonate in non-AKS clusters. This is for providing users with a seamless experience.

For Application Certificates

Only the ABS and AKV integrations support configuring authentication through an Application Certificate at the object level.

For configuring authentication through an Application Certificate for the Bucket API:

Set the .spec.provider field to azure.
Set the .spec.secretRef.name field to the name of the Kubernetes Secret in the same namespace as the Bucket resource.

The Secret must looks like this:

apiVersion: v1
kind: Secret
metadata:
 name: SECRET_NAME
 namespace: NAMESPACE
type: Opaque
stringData:
 tenantId: <tenant ID>
 clientId: <client ID>
 clientCertificate: <certificate and private key>
 # Plus optionally
 clientCertificatePassword: <password if the private key is encrypted>
 clientCertificateSendChain: "true" # this boolean value must be quoted

For SOPS decryption with the Kustomization API:

Set the .spec.decryption.secretRef.name field to the name of the Kubernetes Secret in the same namespace as the Kustomization resource.

The Secret must look like this:

apiVersion: v1
kind: Secret
metadata:
 name: SECRET_NAME
 namespace: NAMESPACE
type: Opaque
stringData:
 sops.azure-kv: |
 tenantId: <tenant ID>
 clientId: <client ID>
 clientCertificate: <certificate and private key>
 # Plus optionally
 clientCertificatePassword: <password if the private key is encrypted>
 clientCertificateSendChain: true # this boolean value must NOT be quoted

At the controller level

At the controller level, regardless if authenticating with Workload Identity Federation or Application Certificates, all Flux resources must have the provider field set to azure according to the rules below.

For all Flux resource kinds except for Kustomization and HelmRelease, set the .spec.provider field to azure and leave .spec.serviceAccountName unset.

For SOPS decryption with the Kustomization API, leave the .spec.decryption.serviceAccountName field unset. There’s no provider field for SOPS decryption.

For remote cluster access with the Kustomization and HelmRelease APIs, set the .data.provider field of the ConfigMap referenced by the .spec.kubeConfig.configMapRef field to azure and leave the .data.serviceAccountName field unset.

The controller-level configuration is described in the following sections.

For Workload Identity Federation

Before following the steps below, make sure to complete the cluster setup described here, and to configure the Kubernetes Service Account as described here.

If the cluster is AKS, the controller Kubernetes Service Account and Deployment must be patched during bootstrap:

The Kubernetes Service Account of the controller must be configured to impersonate a Managed Identity. This is done by adding the azure.workload.identity/client-id and azure.workload.identity/tenant-id annotations.
The label azure.workload.identity/use: "true" must be added to the controller Deployment template metadata. This label is used by the AKS mutating webhook to automatically configure the controller pod during admission to use Workload Identity.

The controller patch should look like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: ServiceAccount
 name: "(some-controller)"
 patch: |
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: controller
 annotations:
 azure.workload.identity/client-id: CLIENT_ID
 azure.workload.identity/tenant-id: TENANT_ID
 - target:
 kind: Deployment
 name: "(some-controller)"
 patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: (some-controller)
 spec:
 template:
 metadata:
 labels:
 azure.workload.identity/use: "true"

If the configuration above is done after bootstrap, restart (delete) the controller for the binding to take effect.

If the cluster is not AKS, the controller Deployment must be patched during bootstrap:

A projected volume must be mounted in the controller Deployment with a Kubernetes Service Account token whose audience is set to api://AzureADTokenExchange.
The environment variables shown below must be set in the controller Deployment.

The controller patch should look like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: "(some-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_TENANT_ID
 value: TENANT_ID
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_CLIENT_ID
 value: CLIENT_ID
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_AUTHORITY_HOST
 value: https://login.microsoftonline.com/
 # or https://login.microsoftonline.us/ for US Gov
 # or https://login.chinacloudapi.cn/ for China
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_FEDERATED_TOKEN_FILE
 value: /var/run/service-account/azure-token
 - op: add
 path: /spec/template/spec/volumes/-
 value:
 name: azure-token
 projected:
 sources:
 - serviceAccountToken:
 audience: api://AzureADTokenExchange
 expirationSeconds: 3600
 path: azure-token
 - op: add
 path: /spec/template/spec/containers/0/volumeMounts/-
 value:
 name: azure-token
 mountPath: /var/run/service-account
 readOnly: true

For Application Certificates

Mount the Kubernetes Secret containing the certificate and private key in the controller Deployment and set the environment variables shown below during bootstrap:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - gotk-components.yaml
 - gotk-sync.yaml
patches:
 - target:
 kind: Deployment
 name: "(some-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_TENANT_ID
 value: TENANT_ID
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_CLIENT_ID
 value: CLIENT_ID
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_CLIENT_CERTIFICATE_PATH
 value: /var/run/app-cert/cert-and-key
 - op: add
 path: /spec/template/spec/volumes/-
 value:
 name: app-cert
 secret:
 secretName: SECRET_NAME
 - op: add
 path: /spec/template/spec/containers/0/volumeMounts/-
 value:
 name: app-cert
 mountPath: /var/run/app-cert/cert-and-key
 subPath: cert-and-key
 readOnly: true

 # ---------------
 # Plus optionally
 # ---------------

 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_CLIENT_CERTIFICATE_PASSWORD
 valueFrom:
 secretKeyRef:
 name: SECRET_NAME
 key: key-password
 - op: add
 path: /spec/template/spec/containers/0/env/-
 value:
 name: AZURE_CLIENT_SEND_CERTIFICATE_CHAIN
 value: "true" # this boolean value must be quoted

The Secret should look like this:

apiVersion: v1
kind: Secret
metadata:
 name: SECRET_NAME
 namespace: flux-system
type: Opaque
stringData:
 cert-and-key: <certificate and private key>
 # Plus optionally
 key-password: <password if the private key is encrypted>

At the node level

Only for the ACR integrations Flux supports authentication at the node level for AKS. This is because users often already have to configure authentication at the node level for AKS to be able to pull container images from ACR in order to start pods. By supporting this authentication method Flux allows users to configure ACR authentication in a single way. See docs.

⚠️ Node level authentication may work for other integrations as well, but Flux only has continuous integration tests for the ACR integration in order to support the specific use case described above.

For node-level authentication to work, the .spec.provider field of the Flux resources must be set to azure.

Supported Azure Clouds

Flux integrations described in this document are supported in Azure Public Cloud (the default environment for most Azure users), in specialized environments such as Azure in China, Azure US Government Cloud and in private clouds.

Private Cloud Configuration

To configure Flux for a private cloud, you must set the AZURE_ENVIRONMENT_FILEPATH environment variable at the controller level. This variable should point to a JSON configuration file mounted into the controller pod that defines the custom Azure endpoints. An example configuration file with the list of custom endpoints supported by Flux is shared below.

{
 "resourceManagerEndpoint": "https://management.core.private/",
 "tokenAudience": "https://management.core.private/",
 "containerRegistryDNSSuffix": "azurecr.private"
}

Flux: Support for cross-cloud

Mon, 01 Jan 0001 00:00:00 +0000

The tables below show the compatibility matrix for Flux running inside a source cluster aiming to get access to resources in a target cloud provider using secret-less authentication.

For each target cloud provider, you can find the authentication docs here:

Authentication at the object level (best for multi-tenant use cases)

Source Cluster Type	AWS	Azure	GCP
AKS	✅	✅	✅
EKS	✅	✅	✅
GKE	✅	✅	✅
Other (e.g. `kind`)	✅	✅	✅

Authentication at the controller level (best for single-tenant use cases)

Source Cluster Type	AWS	Azure	GCP
AKS	✅	✅	✅
EKS	✅	✅	✅
GKE	✅	✅	✅
Other (e.g. `kind`)	✅	✅	✅

Note: Secret-based authentication is supported across all combinations, but should be avoided in favor of secret-less authentication when available. Secrets can be stolen to abuse the permissions granted to the identities they represent, and for public clouds this can be done by simply having Internet access. This requires secrets to be regularly rotated and more security controls to be put in place, like audit logs, secret management tools, etc. Secret-less authentication does not have this problem, as the identity is authenticated using a token that is not stored anywhere and is only valid for a short period of time, usually one hour. It’s much harder to steal an identity this way.

Source cluster setup

When configuring access from a source cluster to a target cloud provider, you need to configure the Issuer URL of the source cluster as an OIDC Provider in the target cloud provider. In this section we show how to obtain the Issuer URL of the source cluster. For instructions on how to create an OIDC provider with this URL in the target cloud provider, go to these links:

Obtaining the Issuer URL of the source cluster

Source cluster setup varies depending on the type of cluster. For details on how to set up the source cluster, see the sections that follow. Once this is done, to obtain the Issuer URL you can use the following command:

kubectl get --raw /.well-known/openid-configuration | jq

Example output for a GKE cluster:

{
 "issuer": "https://container.googleapis.com/v1/projects/<projectID>/locations/<location>/clusters/<name>",
 "jwks_uri": "https://<some-ip-address>:443/openid/v1/jwks",
 "response_types_supported": [
 "id_token"
 ],
 "subject_types_supported": [
 "public"
 ],
 "id_token_signing_alg_values_supported": [
 "RS256"
 ]
}

The Issuer URL is:

https://container.googleapis.com/v1/projects/<projectID>/locations/<location>/clusters/<name>

For managed Kubernetes services from cloud providers

Depending on the type of source cluster, the Issuer URL may already be public and no cluster setup will be required. This is usually the case for the managed Kubernetes services from the cloud providers, e.g. EKS and GKE. AKS needs setup.

For clusters without a built-in public Issuer URL

Some clusters do not have a built-in public Issuer URL, like self-managed, on-premises, or kind clusters running on a local machine or in CI pipelines.

The Kubernetes API Server offers a couple of binary flags that allow customizing the Issuer URL and JWKS URI of the cluster’s service account token issuer. See docs.

These flags are:

--service-account-issuer: Allows setting the issuer field returned in the /.well-known/openid-configuration endpoint as shown in the example above.
--service-account-jwks-uri: Allows setting the jwks_uri field returned in the /.well-known/openid-configuration endpoint as shown in the example above.

This ability to customize the Issuer URL and JWKS URI enables users to host static discovery documents on public URLs, e.g. with CDNs or object storage systems like AWS S3, Azure Blob Storage or GCP Cloud Storage.

Note: This customization is achieved without exposing the Kubernetes API Server endpoint to the Internet. Furthermore, JWKS stands for JSON Web Key Set, which in the OIDC protocol is a JSON-encoded set of public keys that cloud providers can use to verify the signatures of JWTs issued by external identity providers, like the Kubernetes service account token issuer. In other words, the JWKS document content is meant to be public.

⚠️ When performing this operation, remember to set the --service-account-issuer flag with both the new and the previous value, in this specific order, to avoid disruption in the cluster. This flag is allowed to be set multiple times for this purpose. The first value will be used to issue new tokens, and all the values will be used to determine which issuers are accepted by the API Server when authenticating requests, i.e. when validating existing tokens. Bear in mind that the service account tokens are used not only by Flux to get access on cloud providers, but also by Flux and many other core components of Kubernetes to authenticate with the API Server. You need to be extremely cautious performing this operation. Make sure to update the API Server flags only after you have successfully uploaded the discovery documents and confirmed that they are publicly accessible.

Steps to perform this setup:

Run the command kubectl get --raw /.well-known/openid-configuration | jq shown above to get the current discovery document.
Change the issuer field to the new Issuer URL you want to use. For example, if you decide to use AWS S3, the Issuer URL should look like this: https://s3.amazonaws.com/<bucket-name>.
Change the jwks_uri field to the new JWKS URI you want to use. For example, if you decide to use AWS S3, the JWKS URI should look like this: https://s3.amazonaws.com/<bucket-name>/openid/v1/jwks.
Upload the modified discovery document to the public URL you want to use. For example, if you decide to use AWS S3, the object key must be .well-known/openid-configuration and the bucket must be publicly accessible.
Test that the discovery document was correctly uploaded by curling the URL you created. For example, if you decide to use AWS S3, the curl command should look like this: curl https://s3.amazonaws.com/<bucket-name>/.well-known/openid-configuration. The output should be the modified discovery document you uploaded, with the issuer field matching https://s3.amazonaws.com/<bucket-name> and the jwks_uri field matching https://s3.amazonaws.com/<bucket-name>/openid/v1/jwks.
Run the command kubectl get --raw /openid/v1/jwks | jq to get the JWKS document.
Upload the JWKS document to the public URL you want to use. For example, if you decide to use AWS S3, the object key must be openid/v1/jwks and the bucket must be publicly accessible.
Test that the JWKS document was correctly uploaded by curling the URL you created. For example, if you decide to use AWS S3, the curl command should look like this: curl https://s3.amazonaws.com/<bucket-name>/openid/v1/jwks. The output should be the JWKS document you uploaded.
Finally, update the Kubernetes API Server flags mentioned above to use the new Issuer URL and JWKS URI.

Here’s an example of the output of kubectl get --raw /openid/v1/jwks:

{
 "keys": [
 {
 "use": "sig",
 "kty": "RSA",
 "kid": "BfK54vFk8UqizgJVa1BfRfNl-C5c3mWwQ1o_-bA-yAo",
 "alg": "RS256",
 "n": "qN1iYk24aS<... very long redacted public key string ...>IvUY8e4wSaw",
 "e": "AQAB"
 }
 ]
}

Rotating the private key used by Kubernetes to sign service account tokens

If you manage the cluster yourself, you may (probably!) want to rotate the private key used by API Server to issue service account tokens from time to time. When doing so, you must remember to update the JWKS document upstream, otherwise Flux will lose access to your cloud providers.

See how you can manage/rotate the private key used by API Server to sign the service account tokens here. Quote from the docs:

--service-account-signing-key-file string: “Specifies the path to a file that contains the current private key of the service account token issuer. The issuer signs issued ID tokens with this private key.”
--service-account-key-file strings: “Specifies the path to a file containing PEM-encoded X.509 private or public keys (RSA or ECDSA), used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If specified multiple times, tokens signed by any of the specified keys are considered valid by the Kubernetes API Server.”

The ability to specify multiple public keys for signature verification is for allowing rotations to take place without disruption. Existing tokens will be valid until they expire or until the public key that can verify them is removed from the API Server.

⚠️ Bear in mind that service account tokens are used not only by Flux to get access on cloud providers, but also by Flux and many other core components of Kubernetes to authenticate with the API Server. You need to be extremely cautious when rotating this private key.

Special feature from GCP Workload Identity Federation

GCP Workload Identity Federation allows you to directly upload the JWKS document, which frees you from having to customize the API Server flags and from having to host the discovery documents on public URLs, but you still need to update the JWKS document if you rotate the private key of the cluster. See docs.

As of now, GCP is the only cloud provider that supports this feature.

Flux – Integrations

Flux: Amazon Web Services

Identity

Access Management

Granting permissions

Via AWS Controllers for Kubernetes IAM custom resources

Via Terraform/OpenTofu IAM resources

Via the aws CLI

Allowing Kubernetes Service Accounts to assume IAM Roles (Trust Policies)

Authorization

For Amazon Elastic Container Registry

For Amazon Public Elastic Container Registry

For Amazon Simple Storage Service

For Amazon Key Management Service

For Amazon Elastic Kubernetes Service

Permissions for the EKS API

Permissions inside the remote cluster

Authentication

With EKS Pod Identity

With OIDC Federation

Supported clusters

Supported identity types

With IAM User Access Keys

At the object level

For OIDC Federation

For IAM User Access Keys

At the controller level

For OIDC Federation

For IAM User Access Keys

At the node level

Flux: Google Cloud Platform

Identity

Access Management

Granting permissions

To GCP Service Accounts

To Kubernetes Service Accounts

Via Config Connector IAM custom resources

Via Terraform/OpenTofu IAM resources

Via the gcloud CLI

Authorization

For Google Cloud Artifact Registry

For Google Cloud Storage

For Google Cloud Key Management Service

For Google Kubernetes Engine

Permissions for the GKE API

Permissions inside the remote cluster

For Google Cloud Pub/Sub

Authentication

With Workload Identity Federation

Supported clusters

Supported identity types

With GCP Service Account Keys

At the object level

For Workload Identity Federation

For GCP Service Account Keys

At the controller level

For Workload Identity Federation

For GCP Service Account Keys

At the node level

Flux: Microsoft Azure

Identity

Access Management

Granting permissions

Via Azure Service Operator custom resources

Via Terraform/OpenTofu resources

Via the az CLI

Authorization

For Azure Container Registry

For Azure DevOps

Via Terraform/OpenTofu resources

Via the az CLI

For Azure Blob Storage

For Azure Key Vault

For Azure Kubernetes Service

Permissions for the AKS APIs

Permissions inside the remote cluster

For Azure Event Hubs

Authentication

With Workload Identity Federation

Supported clusters

Via the `aws` CLI

Via the `gcloud` CLI

Via the `az` CLI

Via the `az` CLI