Rules에 등록할 Prometheus 쿼리 작성
Adding alert rules
https://docs.syseleven.de/metakube-accelerator/building-blocks/observability-monitoring/kube-prometheus-stack#adding-alert-rules
e.g. GCE CPU 사용률
sum by (instance, instance_name, project_id, zone)(stackdriver_gce_instance_compute_googleapis_com_instance_cpu_utilization)
/ sum by (instance, instance_name, project_id, zone)(stackdriver_gce_instance_compute_googleapis_com_instance_cpu_reserved_cores)
* 100
Rules 등록
kube-prometheus-stack helm chart의 values.yaml에 위에 작성한 쿼리를 등록한다.
- expr: prometheus 쿼리 작성
- annotations을 작성하여 이후 알람에서 원하는 메시지를 받을 수 있다.
Install Prometheus 참고
2022.05.29 - [Monitoring/Prometheus] - [Prometheus]Install prometheus for kubernetes monitoring with helm chart
rule-name 아래에 사용에 맞게 내용을 작성한다
additionalPrometheusRulesMap: #{}
rule-name:
groups:
- name: "GCP-monitoring.rules"
rules:
- alert: GCE CPU over 50 %
for: 1m
expr: sum by (instance, instance_name, project_id, zone)(stackdriver_gce_instance_compute_googleapis_com_instance_cpu_utilization) / sum by (instance, instance_name, project_id, zone)(stackdriver_gce_instance_compute_googleapis_com_instance_cpu_reserved_cores) * 100 > 10
annotations:
message: GCE CPU over 50 % for 2 min
이후 kube-prometheus-stack helm 을 재시작하면 등록된 rule을 확인할 수 있다.
Prometheus, Stackdriver-export 설치 방법
