How do you map secrets from AWS Secret Manager to the EKS cluster?

By Hari Haran CSUpdated as on March 27, 2024 5 minutes read 178 Views

Introduction:

In this blog post, we will provide a step-by-step guide on how to integrate AWS Secret Manager with an AWS Kubernetes cluster. The AWS Secret Manager is a fully-managed service that enables the storage and retrieval of secrets, such as database credentials and API keys. By integrating AWS Secret Manager with an AWS Kubernetes cluster, you can easily manage and secure sensitive data that your applications need to run.

We will also explain how credentials can be injected into a pod, which is a discrete unit of deployment in Kubernetes. With this technique, you can ensure that your applications have the necessary credentials to access other AWS services, such as databases and APIs.

What is CSI Driver?

A CSI driver is a plugin that allows Kubernetes to interact with external storage systems or other storage-related functionalities. When it comes to secrets, CSI drivers enable Kubernetes to securely retrieve sensitive information such as passwords, API keys, or certificates from external secret stores or vaults.

Using CSI drivers for secrets offers several advantages, including:

  1. Enhanced Security: CSI drivers often integrate with secure secret management systems or vaults, ensuring that sensitive information is encrypted and protected at rest and in transit.
  2. Flexibility: CSI drivers provide a flexible mechanism for integrating with various secret management solutions, allowing organizations to use their preferred tool or service.
  3. Scalability: By leveraging CSI drivers, Kubernetes can efficiently manage secrets at scale, ensuring consistent and reliable access to sensitive information across clusters and applications.

Prerequisites:

  • AWS EKS Cluster
  • AWS Secret Manager
  • CLI tools used for this purpose – ekctl, kubectl & aws cli.
  • HELM

Let’s Get Started

Create an EKS cluster and make sure you can run the below commands successfully.

  • list cluster [ $ kubectl cluster-info ]
  • list nodes [ $ kubectl get node ]

Install CSI [ Container Storage interface ] Secret Store driver using helm charts

Install ASCP also known as AWS Secret & Configuration provider

  • kubectl apply -f
    https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Note:

Create secrets in AWS Secret Manager, for demo purposes i have created a username & password as a test-user.

Click next, provide a name for your secret and create your necessary secret.

Create an IAM policy to allow access for pods to talk to the secret manager.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Statement1",
			"Effect": "Allow",
			"Action": [
				"secretsmanager:DescribeSecret",
				"secretsmanager:GetSecretValue"
			],
			"Resource": [
			    "arn:aws:secretsmanager:{Region}:{Account}:secret:{SecretId}"
			]
		}
	]
}

Once IAM policy is created copy it to your clipboard.

Create a service account and attach the policy arn,

eksctl create iamserviceaccount --name test-user-service-account
--region=”$<eks-cluster-region>" --cluster "$<your-cluster-name>"
--attach-policy-arn "$<policy-arn>" --approve --override-existing-serviceaccounts

FYI:

While performing the above command you might end up with the below error,

Issue:

while attaching a policy to cluster, as a non-root user

  • Error: unable to create iamserviceaccount(s) without IAM OIDC provider enabled

Fix

Note:

Amazon EKS supports using OpenID Connect (OIDC) identity providers as a method to authenticate users to your cluster. OIDC identity providers can be used with or as an alternative to AWS Identity and Access Management (IAM)

Below are several ways that you can plan to add secrets to EKS

  • Mount secrets by name or ARN
  • Mount key/value pairs from a secret [ We chose this ]
  • Define a failover Region for a multi-region secret
  • Choose a failover secret to mount

Create a secret provider sp.yaml file eg.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-deployment
  labels:
    app: test-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      serviceAccountName: eks-test-sa
      volumes:
      - name: test-app-secret
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: aws-secrets
      containers:
      - name: test-app
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: test-app-secret
          mountPath: "/mnt/db/secrets"
          readOnly: true

Issue:

when you try to log-in to a container

error:

Internal error occurred: error executing command in container: failed to exec in container: failed to start exec “467c544f0f458d70a6a0ff2ca2d7c2a65f334d681f8757b77c05371d5bdd235f”: OCI runtime exec failed: exec failed: unable to start container process: exec: “C:/Program Files/Git/usr/bin/bash”: stat C:/Program Files/Git/usr/bin/bash: no such file or directory: unknown

Fix

Any updates to secrets in the secret manager will be reflected only when a restart is made against the pod.

Conclusion:

We want you to feel confident that you’re getting the most out of your reading time, and that you’ll leave with a better understanding of the subject matter. So relax, and let us do the heavy lifting for you. With iDevopz you can count on getting the information you need in a friendly, accessible way.

If you found this blog helpful, please feel free to share it with your friends. Thanks for checking it out.