Managing secrets and environment variables

This article mainly focuses on using Bitnami-labs's kubeseal project to manage secrets for applications on the Kubernetes cluster.

Thanks to the guys at Bitnami in making this project. I personally find this very helpful in managing secrets in GitOps way. The secrets are one way sealed so you can safely store them in your GitHub repository. I wrote a small command line tool to easily seal your secrets and check them in the Github repository. We will go through step by step, so let’s get started!

Pre-requisites#

• A running Kubernetes cluster and the ability to run kubectl commands.
• Basic understanding of javascript and nodejs is helpful but you can still follow the steps even if you are not familiar.
• Install nodejs, use brew if you are on mac — brew install node. This should also install npm- the package manager for node.

Installing Kubeseal#

Let’s first look at installing kubeseal on the client side, which will be your laptop or some device where you will seal the secrets.

• You can follow this readme, https://github.com/bitnami-labs/sealed-secrets/blob/master/README.md to install it on both the client-side and on the Kubernetes cluster.
• Otherwise, if you are on macOS you can use brew, brew install kubeseal. This will install the latest version of kubesealwhich is v0.7.0.

Run the following commands to install the controller on the cluster,

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.7.0/controller.yaml
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.7.0/sealedsecret-crd.yaml

This should install sealed-secrets-controller on thekube-systemnamespace.

kubectl get pods -n kube-system, this should list the running pods, check if you can see thesealed-secrets-controller-xxxxxrunning as shown in the bold below.

NAME                                        READY   STATUS    RESTARTS   AGE
aws-node-brlt2                              1/1     Running   0          9d
aws-node-dsmn5                              1/1     Running   0          9d
coredns-6747dfb47f-djpcn                    1/1     Running   0          9d
coredns-6747dfb47f-vrqj5                    1/1     Running   0          9d
kube-proxy-kjcqg                            1/1     Running   0          9d
kube-proxy-lpgjp                            1/1     Running   0          9d
sealed-secrets-controller-6b9f699f5-lvp4g   1/1     Running   0          9d
tiller-deploy-54fc6d9ccc-pphgr              1/1     Running   0          9d

Now, we have to download the public key certificate which will be used to seal the secrets. The controller will use the private key installed on the cluster to decrypt the secrets. This way no unauthorized user can have access to the secrets. You can save these sealed secrets on GitHub along with the public key.

Downloading of the public key depends on how you set up the cluster. In my case, I am running the cluster on AWS EKS. I will provide a few scenarios to download the key. You can also have a look at this GitHub issue to get more idea —https://github.com/bitnami-labs/sealed-secrets/issues/110.

• Usually, kubeseal --fetch-cert > mycert.pem should download the certificate. If you see any errors, try other methods.
• You need cluster admin access to download the certificate. In my case, I have a tiller service account I created as part of the helm installation which has cluster admin access, so I run the following command to first get the token,kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep tiller | awk ‘{print $1’) You should copy the token from the output which will be somewhat as per below screenshot.

Data
====
token:      xxxxxxx

• Run kubeseal --fetch-cert --token xxxxx. This will give an output of the certificate. Save it to a file.
• Another method to get the token is, kubectl port-forward sealed-secrets-controller-xxxxxx 8080:8080 -n kube-system and then run curl -O localhost:8080/v1/cert.pem. This will output the file cert.pem with the certificate.

Sealing secrets#

You can simply use the file now to seal secrets. First, generate a local file with the command -

kubectl create secret generic mysecret --dry-run --from-literal=foo=bar -o json > mysecret.json
kubeseal --cert cert.pem mysecret.json > mysealedsecret.json

We will use these steps to automate the process of creating secrets or environment variables for apps running on the cluster.

Steps to automate#

• For this, you need to clone the repo — https://github.com/dnmahendra/kube-secret-manager
• cd kube-secret-manager && npm i -g ./
• This will install seal-secret command line tool globally.
• The folder structure of this app is,

.
├── apps
│   ├── blaze
│   │   ├── development
│   │   ├── prod
│   │   └── staging
│   └── fenix
│       ├── development
│       ├── prod
│       └── staging
├── bin
└── lib

• Copy the certificate file that was generated before as seal-secrets-prod.pem in the lib folder. You should create each certificate file per stage. This is important as the code inlib/encrypt.js relies on the stages. You can override these stages as per your requirement in lib/values.js file. Also, make sure your app names and stages should match the folder structure under apps directory.
• Run seal-secret encrypt and follow the prompts.
• This will create sealedSecrets.json file under apps/blaze/prod directory.
• The name and namespace used are very important as kubeseal uses them to encrypt the secret. If you don’t specify any namespace then the default will be used.
• You can follow the same procedure to add more secrets to the existing sealedSecrets.json file.
• Now you can run kubectl apply -f apps/blaze/prod/sealedSecrets.json to configure the secrets on the cluster.
• All the code that facilitates this process is in lib/encrypt.js. It’s just a few lines code and easy to understand if you are familiar with javascript.

That’s it! now you can maintain secretes or environment variables for all your apps in the Github repository and keep track of all changes. I hope this helps!