목표
- Google Cloud 영구 디스크(동적으로 만들어진 항목 또는 기존 항목)로 사용할 PersistentVolume(PV) 및 PersistentVolumeClaim(PVC)에 대한 매니페스트 만들기
- Google Cloud 영구 디스크 PVC를 파드의 볼륨으로 마운트하기
- 매니페스트를 사용하여 StatefulSet 만들기
- Google Cloud 영구 디스크 PVC를 StatefulSet의 볼륨으로 마운트하기
- 파드가 중지되었다가 다시 시작될 때 StatefulSet의 파드와 특정 PV의 연결 확인하기
작업 1. PVC, PV 만들기
Cluister API access 설정
gcloud container clusters get-credentials standard-cluster-1 --zone us-central1-a --project [project name]
PVC 생성
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hello-web-disk
spec:
accessModes:
- ReadWriteOnce
storageClassName: standard-rwo
resources:
requests:
storage: 30Gi
apply
$ kubectl apply -f pvc-demo.yaml
persistentvolumeclaim/hello-web-disk created
$ kubectl get persistentvolumeclaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
hello-web-disk Bound pvc-ae061f01-f849-4189-86c7-ed3e14ed5b38 30Gi RWO standard 21s
작업 2. Pod에 Google Cloud Persistent disk PVC 마운트
Pod에 PVC Mount
pod.yaml
- 해당 볼륨을 nginx 컨테이너 내부의 경로 /var/www/html에 마운트
- 테이너 내부에 있는 이 디렉터리에 저장된 파일은 Persistent disk에 저장되며, 파드와 컨테이너가 종료되었다가 다시 만들어지더라도 유지된다.
kind: Pod
apiVersion: v1
metadata:
name: pvc-demo-pod
spec:
containers:
- name: frontend
image: nginx
volumeMounts:
- mountPath: "/var/www/html"
name: pvc-demo-volume
volumes:
- name: pvc-demo-volume
persistentVolumeClaim:
claimName: hello-web-disk
apply
$ kubectl apply -f pod-volume-demo.yaml
pod/pvc-demo-pod created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pvc-demo-pod 1/1 Running 0 83s
pod내 container에서 mount 확인
$ kubectl exec -it pvc-demo-pod -- sh
# echo Test webpage in a persistent
# volume!>/var/www/html/index.html
# chmod +x /var/www/html/index.html#
# cat /var/www/html/index.html
Test webpage in a persistent volume!
# exit
PV의 지속성 테스트하기
- Cluster에서 pod를 삭제하고 PV가 여전히 존재하는지 확인한 다음, pod를 다시 배포하고 PV의 콘텐츠가 그대로 유지되는지 확인.
pod 삭제 및 pvc 확인
$ kubectl delete pod pvc-demo-pod
pod "pvc-demo-pod" deleted
$ kubectl get pods
No resources found in default namespace.
$ kubectl get persistentvolumeclaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
hello-web-disk Bound pvc-ae061f01-f849-4189-86c7-ed3e14ed5b38 30Gi RWO standard 7m2s
pod 재배포
$ kubectl apply -f pod-volume-demo.yaml
pod/pvc-demo-pod created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pvc-demo-pod 1/1 Running 0 17s
pod내 container에서 data 확인
$ kubectl exec -it pvc-demo-pod -- sh
# cat /var/www/html/index.html
Test webpage in a persistent volume!
# exit
작업 3. PVC로 StatefulSet 만들기
- 이 작업에서는 StatefulSet에서 PVC를 사용합니다.
- StatefulSet는 배포와 유사합니다. 단, 파드에 고유한 식별자가 제공됩니다.
PVC 할당 해제하기
StatefulSet 만들기
statefulset-demo.yaml
- service: LoadBalancer
- replica: 3
- nginx 컨테이너 및 이름이 hello-web-disk 인 30GB PVC용 volumeClaimTemplate
- nginx 컨테이너는 이전 작업에서처럼 /var/www/html에 hello-web-disk라는 PVC를 마운트
kind: Service
apiVersion: v1
metadata:
name: statefulset-demo-service
spec:
ports:
- protocol: TCP
port: 80
targetPort: 9376
type: LoadBalancer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-demo
spec:
selector:
matchLabels:
app: MyApp
serviceName: statefulset-demo-service
replicas: 3
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: MyApp
spec:
containers:
- name: stateful-set-container
image: nginx
ports:
- containerPort: 80
name: http
volumeMounts:
- name: hello-web-disk
mountPath: "/var/www/html"
volumeClaimTemplates:
- metadata:
name: hello-web-disk
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 30Gi
apply
$ kubectl apply -f statefulset-demo.yaml
service/statefulset-demo-service created
statefulset.apps/statefulset-demo created
StatefulSet에서 파드의 연결 확인하기
$ kubectl describe statefulset statefulset-demo
Name: statefulset-demo
Namespace: default
CreationTimestamp: Sun, 04 Sep 2022 11:47:32 +0000
Selector: app=MyApp
Labels: <none>
Annotations: <none>
Replicas: 3 desired | 3 total
Update Strategy: RollingUpdate
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=MyApp
Containers:
stateful-set-container:
Image: nginx
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/var/www/html from hello-web-disk (rw)
Volumes: <none>
Volume Claims:
Name: hello-web-disk
StorageClass:
Labels: <none>
Annotations: <none>
Capacity: 30Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 2m19s statefulset-controller create Claim hello-web-disk-statefulset-demo-0 Pod statefulset-demo-0 in StatefulSet statefulset-demo success
Normal SuccessfulCreate 2m19s statefulset-controller create Pod statefulset-demo-0 in StatefulSet statefulset-demo successful
Normal SuccessfulCreate 2m1s statefulset-controller create Claim hello-web-disk-statefulset-demo-1 Pod statefulset-demo-1 in StatefulSet statefulset-demo success
Normal SuccessfulCreate 2m statefulset-controller create Pod statefulset-demo-1 in StatefulSet statefulset-demo successful
Normal SuccessfulCreate 105s statefulset-controller create Claim hello-web-disk-statefulset-demo-2 Pod statefulset-demo-2 in StatefulSet statefulset-demo success
Normal SuccessfulCreate 105s statefulset-controller create Pod statefulset-demo-2 in StatefulSet statefulset-demo successful
pod, pvc 확인
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
statefulset-demo-0 1/1 Running 0 3m29s
statefulset-demo-1 1/1 Running 0 3m10s
statefulset-demo-2 1/1 Running 0 2m55s
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
hello-web-disk Bound pvc-ae061f01-f849-4189-86c7-ed3e14ed5b38 30Gi RWO standard 23m
hello-web-disk-statefulset-demo-0 Bound pvc-115ec773-ab0d-47b4-a532-c3466ca8355a 30Gi RWO standard 3m35s
hello-web-disk-statefulset-demo-1 Bound pvc-7330f634-5f8d-4885-b4a5-2a065307d45e 30Gi RWO standard 3m17s
hello-web-disk-statefulset-demo-2 Bound pvc-75f9379d-57ab-4746-88b3-36cb4ab7879a 30Gi RWO standard 3m1s
생성된 첫 번째 pvc 세부정보 확인
$ kubectl describe pvc hello-web-disk-statefulset-demo-0
Name: hello-web-disk-statefulset-demo-0
Namespace: default
StorageClass: standard
Status: Bound
Volume: pvc-115ec773-ab0d-47b4-a532-c3466ca8355a
Labels: app=MyApp
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 30Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: statefulset-demo-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 4m12s pd.csi.storage.gke.io_gke-c5f582a493a242ff8e0e-4ae8-2c45-vm_7254ad77-5f97-4b8e-b6ca-6aa849398702 External provisioner is provisioning volume for claim "default/hello-web-disk-statefulset-demo-0"
Normal ExternalProvisioning 4m9s (x2 over 4m12s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
Normal ProvisioningSucceeded 4m8s pd.csi.storage.gke.io_gke-c5f582a493a242ff8e0e-4ae8-2c45-vm_7254ad77-5f97-4b8e-b6ca-6aa849398702 Successfully provisioned volume pvc-115ec773-ab0d-47b4-a532-c3466ca8355a
작업 4. StatefulSet가 관리하는 파드에 대한 영구 볼륨 연결의 지속성 확인하기
pod, demo-o에서 pvc 연결 지속성 확인
$ kubectl exec -it statefulset-demo-0 -- sh
# cat /var/www/html/index.html
cat: /var/www/html/index.html: No such file or directory
# echo Test webpage in a persistent
# volume!>/var/www/html/index.html
# chmod +x /var/www/html/index.html#
# cat /var/www/html/index.html
Test webpage in a persistent volume!
# exit
demo-o pod 삭제
$ kubectl delete pod statefulset-demo-0
pod "statefulset-demo-0" deleted
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
statefulset-demo-0 0/1 ContainerCreating 0 5s
statefulset-demo-1 1/1 Running 0 6m37s
statefulset-demo-2 1/1 Running 0 6m22s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
statefulset-demo-0 1/1 Running 0 14s
statefulset-demo-1 1/1 Running 0 6m46s
statefulset-demo-2 1/1 Running 0 6m31s
데이터 확인
$ kubectl exec -it statefulset-demo-0 -- sh
# cat /var/www/html/index.html
Test webpage in a persistent volume!
# exit
- 다른 StatefuleSet에서는 데이터가 없다
$ kubectl exec -it statefulset-demo-1 -- sh
# cat /var/www/html/index.html
cat: /var/www/html/index.html: No such file or directory
# exit
참고
dynamic_provisioning 프로비저닝 storageclass 차이
Name: premium-rwo
IsDefaultClass: No
Annotations: components.gke.io/component-name=pdcsi,components.gke.io/component-version=0.11.8,components.gke.io/layer=addon
Provisioner: pd.csi.storage.gke.io
Parameters: type=pd-ssd
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
Events: <none>
Name: standard
IsDefaultClass: Yes
Annotations: storageclass.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/gce-pd
Parameters: type=pd-standard
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
Name: standard-rwo
IsDefaultClass: No
Annotations: components.gke.io/layer=addon,storageclass.kubernetes.io/is-default-class=false
Provisioner: pd.csi.storage.gke.io
Parameters: type=pd-balanced
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
Events: <none>
resize pvc
down size는 불가능
- The PersistentVolumeClaim "hello-web-disk" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value
up size log
$ k describe pvc hello-web-disk
W0905 13:08:25.162928 21311 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult <https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke>
Name: hello-web-disk
Namespace: monitoring
StorageClass: standard-rwo
Status: Bound
Volume: pvc-d3e360e2-aecb-4a4d-8d08-5288be5ccf44
Labels:
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
volume.kubernetes.io/selected-node: gke-cms3-cluster-cms3-node-1-9bb08beb-gv6t
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 35Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: pvc-demo-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 4m1s (x5 over 4m49s) persistentvolume-controller waiting for first consumer to be created before binding
Normal ExternalProvisioning 3m55s persistentvolume-controller waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
Normal Provisioning 3m55s pd.csi.storage.gke.io_gke-c8d0c59d0cce40de995d-b068-b620-vm_66037b56-cb0a-48ea-8818-026ed678a78b External provisioner is provisioning volume for claim "monitoring/hello-web-disk"
Normal ProvisioningSucceeded 3m50s pd.csi.storage.gke.io_gke-c8d0c59d0cce40de995d-b068-b620-vm_66037b56-cb0a-48ea-8818-026ed678a78b Successfully provisioned volume pvc-d3e360e2-aecb-4a4d-8d08-5288be5ccf44
Warning ExternalExpanding 25s volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
Normal Resizing 25s external-resizer pd.csi.storage.gke.io External resizer is resizing volume pvc-d3e360e2-aecb-4a4d-8d08-5288be5ccf44
Normal FileSystemResizeRequired 16s external-resizer pd.csi.storage.gke.io Require file system resize of volume on node
Normal FileSystemResizeSuccessful 8s kubelet MountVolume.NodeExpandVolume succeeded for volume "pvc-d3e360e2-aecb-4a4d-8d08-5288be5ccf44"
Google Cloud Study Jam을 진행하며 학습한 내용의 정리입니다.
https://sites.google.com/view/studyjam-kr/home?authuser=0
