[Kubernetes]Drain and Delete for node change in Kubernetes

Purpose

서버의 물리적인 자원교체, 버전 업그레이드를 위해 클러스터에서 노드를 제외시키기 위함.

Work process

1. 노드 제외
   1. 노드를 클러스터에서 제외시킨다
   2. 노드를 리셋시킨다 (제외시킨 노드에서)
2. 노드 추가
   1. Join을 위한 토큰값을 얻는다
   2. 얻은 토큰으로 클러스터에 Join

---

 

1. Node drain & delete

## drain
root@AJTV005 [~]kubectl drain ajtv009 --ignore-daemonsets
node/ajtv009 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-9bl98, kube-system/weave-net-bfs2f
evicting pod default/deploytest-79bdb557f6-fpl8c
pod/deploytest-79bdb557f6-fpl8c evicted
node/ajtv009 evicted

root@AJTV005 [~]kubectl get nodes -o wide
NAME      STATUS                     ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
ajtv005   Ready                      master   22h   v1.19.4   10.50.107.21   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv006   Ready                      master   22h   v1.19.4   10.50.107.22   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv007   Ready                      <none>   22h   v1.19.4   10.50.107.24   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv008   Ready                      <none>   22h   v1.19.4   10.50.107.25   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv009   Ready,SchedulingDisabled   master   22h   v1.19.4   10.50.107.26   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14

## delete node
root@AJTV005 [~]kubectl delete node ajtv009
node "ajtv009" deleted

root@AJTV005 [~]kubectl get node -o wide
NAME      STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
ajtv005   Ready    master   22h   v1.19.4   10.50.107.21   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv006   Ready    master   22h   v1.19.4   10.50.107.22   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv007   Ready    <none>   22h   v1.19.4   10.50.107.24   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14
ajtv008   Ready    <none>   22h   v1.19.4   10.50.107.25   <none>        CentOS Linux 7 (Core)   3.10.0-1160.6.1.el7.x86_64   docker://19.3.14

---

 

2. kubeadm reset (삭제한 노드에서)

kubeadm reset

rm -rf /etc/cni/net.d
rm -rf $HOME/.kube/config

---

 

3. Get token, certs and hash key

클러스터에 Join하기 위해 아래의 3가지 토큰을 구한다

3.1 Create token

root@AJTV005 [~]kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
v4f4is.ss7k5e1t27kgc46u   1h          2020-12-08T17:17:40+09:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
root@AJTV005 [~]kubeadm token delete v4f4is.ss7k5e1t27kgc46u
bootstrap token "v4f4is" deleted

root@AJTV005 [~]kubeadm token create
W1208 15:46:28.364740   28648 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
uljmut.h6sy5ibklt0d9vuh
root@AJTV005 [~]kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
uljmut.h6sy5ibklt0d9vuh   23h         2020-12-09T15:46:28+09:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token

token TTL 24h

3.2 Get The CA key hash

root@AJTV005 [~/scripts]openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a

3.3 인증서 생성 & 업로드

인증서는 아래 두 가지 방법으로 업로드 할 수 있다.
인증서 생성 후 업로드 or 임의의값으로 생성 후 업로드

3.3.1 인증서 인증서 생성 & 업로드

## 인증서 확인
kubeadm alpha certs check-expiration

## 인증서 생성
root@AJTV005 [~]kubeadm alpha certs certificate-key
1fa779fa84a83eb6cc7f48e817928eff5690a06b3d4cc11682480e364b913091

## 인증서 업로드 
kubeadm init phase upload-certs --upload-certs --certificate-key=1fa779fa84a83eb6cc7f48e817928eff5690a06b3d4cc11682480e364b913091

TTL 2H

3.3.2 임의의값으로 인증서 생성 & 업로드

root@AJTV005 [~/scripts]kubeadm init phase upload-certs --upload-certs
W1209 00:57:53.829050    8487 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
12d076d4733416c12c491ac54f929d43b4b0f721b044d0202b291d20f1656b96

---

4. Join with token

For Master

kubeadm join 10.50.107.23:8443 --token uljmut.h6sy5ibklt0d9vuh --discovery-token-ca-cert-hash sha256:0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a     --control-plane --certificate-key 1fa779fa84a83eb6cc7f48e817928eff5690a06b3d4cc11682480e364b913091 --v=5

kubeadm join 10.50.107.23:8443 --token hbqmn6.4bu4lp8ik046qy78 \
--discovery-token-ca-cert-hash sha256:0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a \
--control-plane \
--certificate-key 07a03068518c444d582123cb0fe38ae217cabbca521d8590667e8ab64322a8a9

For Node

kubeadm join 10.50.107.23:8443 --token uljmut.h6sy5ibklt0d9vuh     --discovery-token-ca-cert-hash sha256:0f63b526211dacb46d50157ce99b53d6bfdc7246551e39baa7b47e396f97542a

---

 

Troubleshooting

Remove master node from a HA Cluster and also from cluster

아래 작업 없이는 HA 구성된 Master Node 교체가 안됨 ( Worker Node는 상관 없음 )

Get member list

## ETCDCTL_AP3 etcdctl member list 명령어로도 획득 가능

root@AJTV005 [/]kubectl exec etcd-ajtv005 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list
38e227bede457131, started, ajtv009, https://10.50.107.26:2380, https://10.50.107.26:2379, false
5f05adedb10bbff4, started, ajtv006, https://10.50.107.22:2380, https://10.50.107.22:2379, false
a8e5615362288545, started, ajtv005, https://10.50.107.21:2380, https://10.50.107.21:2379, false

Remove member

root@AJTV005 [/]kubectl exec etcd-ajtv005 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member remove 38e227bede457131
Member 38e227bede457131 removed from cluster ee7be35e4ed61075