[Kubernetes]Sealed secrets - A Kubernetes controller and tool for one-way encrypted Secrets

https://github.com/bitnami-labs/sealed-secrets

개요

kubernetes secrets config를 git에 저장하려고 한다. 문제는 secrets config는 base64 decode를 이용해 쉽게 해독이 가능하다.

SealedSecret를 이용하면 이 대상 클러스터에서 실행되는 컨트롤러에 의해서만 secrets이 해독될 수 있고 다른 누구도(심지어 원래 작성자도) SealedSecret에서 원래 비밀을 얻을 수 없다.

방법

1. SealedSecret 를 이용하여 암호된 SealedSecret.yaml을 생성한다.
2. SealedSecret.yaml를 배포하여 kubernetes secrets을 생성한다.
3. 주기적으로 Sealed secrets, secrets(e.g. database password)을 변경하고 배포한다.

설치

https://github.com/bitnami-labs/sealed-secrets/releases

Controller (cluster)

복호화를 수행하는 SealedSecret Controller

# add repo
➜ helm repo add sealed-secrets <https://bitnami-labs.github.io/sealed-secrets>
# download helm cart
➜ helm fetch sealed-secrets/sealed-secrets
# install helm chart
➜ helm install sealed-secrets ./ -n kube-system

Kubeseal (client)

secret을 sealed-secrets으로 암호화하는 도구

macos

brew install kubeseal

linux

wget <https://github.com/bitnami-labs/sealed-secrets/releases/download/>/kubeseal--linux-amd64.tar.gz
tar -xvzf kubeseal--linux-amd64.tar.gz kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Usage

SealedSecret and Secret must have the same namespace and name.

controller의 인증서를 사용하는 방법

create basic secrets from yaml

kubectl create secret generic secret-from-sealedsecret --dry-run=client --from-literal=hello=jeff -o yaml > mysecret.yaml

➜ cat mysecret.yaml                                         
apiVersion: v1
data:
  hello: amVmZg==
kind: Secret
metadata:
  creationTimestamp: null
  name: secret-from-sealedsecret

➜ echo -n "amVmZg==" | base64 --decode                              
jeff%

create SealedSecret with above secrets.yaml

cat mysecret.yaml | kubeseal -o yaml > mysealedsecret.yaml

➜ cat mysealedsecret.yaml                                         
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: secret-from-sealedsecret
  namespace: devops
spec:
  encryptedData:
    hello: AgDkLtR/GCM7jGn56NFZpIWJlX4JQx7dnMkPX9BKQdSrk+YaoiS5wkC8aJjSnJZpkPZnXn91YnDnMaNmiH1PBIDt3C25aenLVjRhkRfb+tzBqOUPi1lIN0ZQJTDuUY9EBRyrc1TVC9DDbGOORKAKqFyBxyBCyPNikEudwo46Ra+VWpPylCD6hdeNvibWLhQbxgCsZG+O10KLeuDNhFrddI/MbtqE5iW5uTZ3rc5oeGJRkzCyPaZOLe6hp+4nYF5qEsdJn5Tbp7QXMl+FK9DT0uzB4f+k55yd79IR+IUDBHX5qTI2SjhUecwXS0AJ+UNKj4gNOAqZL0JgImDj+Cn3riqa5eGzhKBi/mt5LHQAXbTsQmDl/OPV9LAPjoNMOJbQZ/Xg5Hr9enQaDgaQH5Arz1V+zC/BPHgaBxNez3wBdQZqQvKuIadeu+4ipab0bLkb8SLnjsgo4lES2g17qNsLlEJdVPacv2bg10N/5EGp70eF4RXj5RJ1C/No+v68wrB80hr+6tmP+2ddq00DnZODjRk8ZJWhj0womAe1b5+tOTbNmpWhTYVirhkMqttouoW8Ql5AqYgE63gvFobiJFB0kBP9Cpw3xzOvoKno/7qmwALgsZ8m5NLdwQ+Q12vvcCJc5XBFVESLwT+LiCfD8sFKmMwcbHgpHB3lGqdOv7ilWfePsu0Scor1Jw6zZ5FbjIS/K6ugZs4P
  template:
    metadata:
      creationTimestamp: null
      name: secret-from-sealedsecret
      namespace: devops

create SealedSecret at once

kubectl create secret generic secret-from-sealedsecret --dry-run=client --from-literal=hello=jeff -o yaml | \\
kubeseal \\
      --controller-name=sealed-secrets-controller \\
      --controller-namespace=kube-system \\
			--format yaml > mysealedsecret.yaml

apply SealedSecret

➜ k apply -f mysealedsecret.yaml                                                                  
sealedsecret.bitnami.com/secret-name created

➜ kubectl get sealedsecret
NAME          STATUS   SYNCED   AGE
secret-name            True     3m48s

➜ k get sealedsecret secret-name -o yaml | k neat 
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: secret-from-sealedsecret
  namespace: devops
spec:
  encryptedData:
    hello: AgB1Fgg5pJnRezpptCZF7RYeGo42FlfEaDW3hGR7B76TJITt1qK0gi+QKj1HZauyHKL5eTF+u1xNC2Cuko3+GbVLQN0HN71w9M/Jvmnzr+OfDlYBSm5AHaWlQkehY9ItnkrupOMj/iHl59e6N3BUHhrKej31j69EOf4comCklJxdzSI/XlOB0j0Xap7Kww8yiGZ9x8K89iVteYIbxUtAjZG2qwyrnk78w3kP6vjYfJuOflZYXc5GSwNLqcCQeN53RhY3EPrykRDD55hoPgdCtzU+SXVcA5CpMPKoPjCJXfalMcOKZE2xDY2B8SxMsJNuwz2Mk4SeWSlE43Gg7pQLNCo7I9zTE2JJmuZrUqyLKLfgiTSJsC4kA/7al3iFAdaRQcgogTwNh21glqX3kh8ZAcL+gYM5uwm9aLKvFv7Fk5DGQWdDp836dCIL9aOR82RDxSyTnrMfD6d12gs39xsJcpRWZyQ0AYLv7LE4/jyHOUeLKHIe61LRkJ2V9h7cW5i1FN++m3CDl77qrihBHEhgSuP+gr/y/w4rHbttTYyknekwLulEhyRM3Sg9m3jbz1bU78By2oAl+kr6rpL0PVSimR73e7Zk2fSmKVlASbnK23bG4je8iM5x1uysiuGglJ0ncmkUwmVFnp9akFVGeDUiqgy8D0wTHwSpjZEQZwRWOSTv0f/1ZJHZwyD80SxV/nRuZ1knatM0
  template:
    metadata:
      creationTimestamp: null
      name: secret-from-sealedsecret
      namespace: devops

➜ k get secrets                               
NAME                                   TYPE                 DATA   AGE
secret-from-sealedsecret               Opaque               1      5m7s

➜ kubectl get secret secret-from-sealedsecret -n devops -o jsonpath='{.data.hello}' | base64 -d
jeff%

edit secrets data and deploy with sealedsecret

kubectl create secret generic secret-from-sealedsecret --dry-run=client --from-literal=hello=jmhan -o yaml | \\
kubeseal \\
      --controller-name=sealed-secrets-controller \\
      --controller-namespace=kube-system \\
			--format yaml > mysealedsecret2.yaml

➜ k apply -f mysealedsecret2.yaml                   
sealedsecret.bitnami.com/secret-from-sealedsecret configured

➜ kubectl get secret secret-from-sealedsecret -n devops -o jsonpath='{.data.hello}' | base64 -d
jmhan%

jeff → jmhan으로 secret data가 변경된걸 알 수 있다.

local에 controller 인증서를 저장하고 사용하는 방법

to retrieve the public cert used for encryption and store it locally. You can then run 'kubeseal --cert mycert.pem' instead to use the local cert

fetch cert from controller

➜ kubeseal \\
--controller-name=sealed-secrets-controller \\
--controller-namespace=kube-system \\
--fetch-cert > mycert.pem

➜ cat mycert.pem             
-----BEGIN CERTIFICATE-----
MIIEzDCCArSgAwIBAgIQJ74eoKgXj1taIh3BQKMJxzANBgkqhkiG9w0BAQsFADAA
MB4XDTIzMDcyMzA3NDkyNFoXDTMzMDcyMDA3NDkyNFowADCCAiIwDQYJKoZIhvcN
-----END CERTIFICATE-----

create SealedSecret with local cert

➜ kubectl create secret generic secret-name --dry-run=client --from-literal=foo=bar -o yaml | \\
kubeseal \\
--controller-name=sealed-secrets-controller \\
--controller-namespace=kube-system \\
--format yaml --cert mycert.pem > mysealedsecret.yaml

➜ kubectl create -f mysealedsecret.yaml

flag

--scope

• strict (default): the secret must be sealed with exactly the same name and namespace. These attributes become part of the encrypted data and thus changing name and/or namespace would lead to "decryption error".
• namespace-wide: you can freely rename the sealed secret within a given namespace.
• cluster-wide: the secret can be unsealed in any namespace and can be given any name.

kubeseal --scope cluster-wide <secret.yaml >sealed-secret.json

Secret Rotation

You should rotate your secrets. default: 720h30m

It is not a form of key rotation. Sealed secrets are not automatically rotated and old keys are not deleted when new keys are generated.

recommendation

주기적으로 모든 secrets(e.g. change the password)을 교체하고, 그 새로운 secrets으로 새로운SealedSecret 을 만들어야 한다.

Re-encryption

오래된 encrypt key를 제거하기 전에 SealedSecrets를 최신 개인 키로 다시 암호화해야 합니다.

kubeseal --re-encrypt <my_sealed_secret.json >tmp.json \\
  && mv tmp.json my_sealed_secret.json

The invocation above will produce a new sealed secret file freshly encrypted with the latest key, without making the secrets leave the cluster to the client. You can then save that file in your version control system (kubeseal --re-encrypt doesn't update the in-cluster object).

backup sealed-secrets

backup of the encryption private keys

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >main.key

echo "---" >> main.key
kubectl get secret -n kube-system sealed-secrets-key -o yaml >>main.key

restore from a backup after some disaster

kubectl apply -f main.key
kubectl delete pod -n kube-system -l name=sealed-secrets-controller