Kubernetes external secrets

In this post we will talk about Kubernetes external secrets. It’s a project developed by the GoDaddy Engeneering Team that allows to use external secrets management systems to securely add secrets to your kubernetes cluster. At the time of writing, it supports the following backends:

• AWS Secrets Manager
• AWS System Manager
• GCP Secret Manager
• Azure Key Vault
• Hashicorp Vault

With external secrets you will be able to use secrets created and stored in both secrets management systems as a centralized place to store all the secrets you need for your applications, CICD, etc. With this approach you will avoid to store secrets in different places and put sensitive data in your code repositories.

Figure 1 description:

1. ExternalSecrets controller is added in the k8s cluster
2. Controller fetches ExternalSecrets using the Kubernetes API
3. Controller uses ExternalSecrets to fetch secret data from external providers (e.g, AWS Secrets Manager)
4. Controller upsert Secrets and pods can access Secrets in their namespaces normally

We use this software in different customers and it works flawlessly.

Now, we will show you an example using AWS cloud infrastructure (EKS) and you will see how easy is to implement this service.

Secret creation

First of all we will create the secret itself. For this example, we will use AWS Secrets Manager and we will create the secret using Cloudformation:

AWSTemplateFormatVersion: '2010-09-09'
Description: Secrets definitions

Parameters:
  EnvName:
    Description: 'Environment name'
    Type: String

Resources:
  DBCredentials:
    Type: 'AWS::SecretsManager::Secret'
    Properties:
      Name: !Sub DB-${EnvName}
      Description:
        Fn::Join:
        - ' - '
        - - 'DB credentials'
          - !Ref EnvName
      SecretString: '
        {
          "user": "myappuser",
          "password": "changeme"
        } '

In this example we’ve created the secret with password value ‘changeme’ and you must change it after secret resource creation. Remember that this yaml file will be stored in your preferred Git provider and we don’t want to store passwords in code repositories! 😉

TIP: You can change the password manually via AWS Console for the first time and after that, for instance, you can implement a password automatic rotation. In this case, make sure that the password configured in your service also changes! For AWS RDS password rotation (that we used in this example) please read this.

Deployment

Once we’ve created the secret with the correct password value, we will install external secrets in our EKS cluster. We will install it via helm package as follows:

# First of all, we add external secrets repo
helm repo add external-secrets https://godaddy.github.io/kubernetes-external-secrets/

# And we install the helm package. In this example, in kube-system namespace but you can install it where you want
helm upgrade --install 
    --namespace kube-system 
    external-secrets external-secrets/kubernetes-external-secrets

Probably, you’ll need to configure the helm chart with appropriate values, please see helm chart docs.

We can check that the pod is running:

$ kubectl get pod external-secrets-598f9cb66f-m5xn5 -n kube-system
NAME                               READY  STATUS   RESTARTS  AGE
external-secrets-598f9cb66f-m5xn5  1/1    Running  0         97d

External secrets object creation

Now it’s time to create the ExternalSecret object:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: db-app-credentials
  namespace: your_namespace
spec:
  backendType: secretsManager
  # optional: specify role to assume when retrieving the data
  roleArn: arn:aws:iam::123456789012:role/secrets-readonly-example
  data:
    - key: DB-
      name: user
    - key: DB-
      name: password

Because we are running k8s in EKS, please note that we used a roleArn that allows access to AWS Secrets Manager. You can do it also granting access to your EKS worker nodes defining and configuring a node instance role.

Moreover, if your k8s cluster runs outside AWS, GCP or Azure you can pass the credentials as environment variables when you deploy the helm chart (please, take a look at the values.yaml file options).

Once we’ve created the ExternalSecret object, after a few seconds, a k8s secret will be created in the specified namespace:

$ kubectl get secret db-app-credentials -n your_namespace 
NAME                            TYPE    DATA     AGE
db-app-credentials              Opaque  2        23m

And the content of the secret will be similar to this:

$ kubectl get secret db-app-credentials-n your_namespace -o yaml
apiVersion: v1
data:
  password: sgdhfh7w98JHGFsq1shdgA==
  user: bXlhcHB1c2VyCg==
kind: Secret
metadata:
  creationTimestamp: "2020-03-30T10:26:34Z"
  name: db-app-credentials
  namespace: your_namespace
  ownerReferences:
  - apiVersion: kubernetes-client.io/v1
    controller: true
    kind: ExternalSecret
    name: db-app-credentials
    uid: 8a69iac3-2007-11ea-ae84-0awqa0e278d8
  resourceVersion: "14684195"
  selfLink: /api/v1/namespaces/your_namespace/secrets/db-app-credentials
  uid: aeha168b-2007-11ea-ae84-0a0ath7278d8
type: Opaque

Now, every time you update your secret in the backend, your secret in k8s cluster will also be updated.

Final notes

You can configure the poll interval (how often external secrets will ask to backend for changes or new secrets). Please, take into account that many cloud providers charge for secret access or api calls. 😉

It’s important to know that Kubernetes external secrets only makes upserts. It will not delete any secret in your k8s cluster in case of failure or backend access/synchronization problems.

That’s it!! 🙂 For more info you can visit the external secrets github repo.

I hope you’ve enjoyed this post and I encourage you to check our blog for other posts that you might find helpful. Do not hesitate to contact us if you would like us to help you on your projects.

See you on the next post!