您可以通過多集群統一報警管理能力,在Fleet實例中配置或修改報警規則。由Fleet實例將報警規則統一下發到指定的關聯集群中,且保證已下發的報警規則在各個關聯集群中的一致性。同時,Fleet實例可以對新關聯的集群自動同步報警規則。本文介紹如何實現艦隊中多集群的統一報警管理。
前提條件
已開啟艦隊管理功能。具體操作,請參見開啟艦隊管理功能。
艦隊的Fleet實例已添加2個關聯集群(服務提供者集群、服務消費者集群)。具體操作,請參見添加關聯集群。
已從ACK One控制臺獲取Fleet實例的KubeConfig,并通過kubectl連接至Fleet實例。
艦隊的Fleet實例已添加2個關聯集群(服務提供者集群、服務消費者集群)。具體操作,請參見添加關聯集群。
在需要統一報警管理的集群上,已安裝報警管理相關組件。具體操作,請參見組件安裝與升級。
背景信息
在多集群場景中,往往所有集群的報警規則是一致的,當需要修改某一個規則時,需要登錄所有集群的控制臺逐一修改,操作繁瑣而且容易出錯。您可以通過多集群統一報警管理能力,在Fleet實例上統一配置報警規則,指定哪些異常需要告警和告警相關的聯系人。更多信息,請參見容器服務報警管理。多集群管理統一報警架構如下圖所示:
步驟一:創建報警聯系人與聯系人分組
您可以通過以下步驟創建報警聯系人和聯系人分組,報警聯系人和聯系人分組創建一次即可,在所有容器服務集群內共享。
登錄容器服務管理控制臺,在左側導航欄選擇集群。
在集群列表頁面中,單擊目標集群名稱,然后在左側導航欄中,選擇 。
說明首次登錄需根據頁面提示進行安裝,若顯示組件未升級,需單擊開始升級完成升級后,進入報警配置頁面。
在報警配置頁面,按照以下步驟完成聯系人創建和聯系人分組創建。
單擊聯系人管理頁簽,然后單擊創建。
在創建聯系人頁面,輸入姓名、電話和郵箱。然后單擊確定。
聯系人創建完成后,您將會收到驗證激活短信或驗證激活郵件,請按相應提示進行激活操作。
單擊聯系人分組管理頁簽,然后單擊創建。
在創建分組頁面,輸入分組名稱,然后選擇分組聯系人,最后單擊確定。
選擇分組聯系人時,將可選聯系人添加到已選聯系人列表,也可移除已選聯系人。
步驟二:獲取報警聯系人分組ID
使用如下Aliyun CLI查詢聯系人分組,獲取其在其他云服務中的內部ID,用于后續配置報警規則。
aliyun cs GET /alert/contact_groups { "contact_groups": [ { "ali_uid": 14783****, "binding_info": "{\"sls_id\":\"ack_14783****_***\",\"cms_contact_group_name\":\"ack_Default Contact Group\",\"arms_id\":\"1****\"}", "contacts": null, "created": "2021-07-21T12:18:34+08:00", "group_contact_ids": [ 2*** ], "group_name": "Default Contact Group", "id": 3***, "updated": "2022-09-19T19:23:57+08:00" } ], "page_info": { "page_number": 1, "page_size": 100, "total_count": 1 } }
在查詢結果信息中提取信息,構建contactGroups。
contactGroups: - arms_contact_group_id: "1****" #從上步查詢結果的contact_groups.binding_info.arms_id獲取。 cms_contact_group_name: ack_Default Contact Group #從上步查詢結果的contact_groups.binding_info.cms_contact_group_name獲取。 id: "3***" #從上步查詢結果的contact_groups.id獲取。
步驟三:創建報警規則
您可以使用如下模板創建報警規則,模板中預置了所有容器服務ACK支持的報警規則,下面以開啟error-events報警規則為例說明報警規則開啟步驟。
報警規則的名稱必須為default,命名空間必須為kube-system。詳細的規則描述,請參見默認報警規則模板。
您在Fleet實例中創建報警規則后,報警實際并未生效,還需要創建分發規則將報警規則分發到關聯集群中,使得報警規則在各關聯集群中生效。
修改error-events報警規則對應的
rules.enable
為enable
。添加從上一步生成的contactGroups字段。將修改后的報警規則模板另存為ackalertrule.yaml。
執行命令
kubectl apply -f ackalertrule.yaml
,在Fleet實例中創建報警規則。
報警規則模板如下:
apiVersion: alert.alibabacloud.com/v1beta1
kind: AckAlertRule
metadata:
name: default
namespace: kube-system
spec:
groups:
- name: error-events
rules:
- enable: enable
contactGroups:
- arms_contact_group_id: "1****"
cms_contact_group_name: ack_Default Contact Group
id: "3***"
expression: sls.app.ack.error
name: error-event
notification:
message: kubernetes cluster error event.
type: event
- name: warn-events
rules:
- enable: disable
expression: sls.app.ack.warn
name: warn-event
notification:
message: kubernetes cluster warn event.
type: event
- name: cluster-core-error
rules:
- enable: disable
expression: prom.apiserver.notHealthy.down
name: apiserver-unhealthy
notification:
message: "Cluster APIServer not healthy. \nPromQL: ((sum(up{job=\"apiserver\"})
<= 0) or (absent(sum(up{job=\"apiserver\"})))) > 0"
type: metric-prometheus
- enable: disable
expression: prom.etcd.notHealthy.down
name: etcd-unhealthy
notification:
message: "Cluster ETCD not healthy. \nPromQL: ((sum(up{job=\"etcd\"}) <= 0)
or (absent(sum(up{job=\"etcd\"})))) > 0"
type: metric-prometheus
- enable: disable
expression: prom.scheduler.notHealthy.down
name: scheduler-unhealthy
notification:
message: "Cluster Scheduler not healthy. \nPromQL: ((sum(up{job=\"ack-scheduler\"})
<= 0) or (absent(sum(up{job=\"ack-scheduler\"})))) > 0"
type: metric-prometheus
- enable: disable
expression: prom.kcm.notHealthy.down
name: kcm-unhealthy
notification:
message: "Custer kube-controller-manager not healthy. \nPromQL: ((sum(up{job=\"ack-kube-controller-manager\"})
<= 0) or (absent(sum(up{job=\"ack-kube-controller-manager\"})))) > 0"
type: metric-prometheus
- enable: disable
expression: prom.ccm.notHealthy.down
name: ccm-unhealthy
notification:
message: "Cluster cloud-controller-manager not healthy. \nPromQL: ((sum(up{job=\"ack-cloud-controller-manager\"})
<= 0) or (absent(sum(up{job=\"ack-cloud-controller-manager\"})))) > 0"
type: metric-prometheus
- enable: disable
expression: prom.coredns.notHealthy.requestdown
name: coredns-unhealthy-requestdown
notification:
message: "Cluster CoreDNS not healthy, continuously request down. \nPromQL:
(sum(rate(coredns_dns_request_count_total{}[1m]))by(server,zone)<=0) or
(sum(rate(coredns_dns_requests_total{}[1m]))by(server,zone)<=0)"
type: metric-prometheus
- enable: disable
expression: prom.coredns.notHealthy.panic
name: coredns-unhealthy-panic
notification:
message: "Cluster CoreDNS not healthy, continuously panic. \nPromQL: sum(rate(coredns_panic_count_total{}[3m]))
> 0"
type: metric-prometheus
- enable: disable
expression: prom.ingress.request.errorRateHigh
name: ingress-err-request
notification:
message: Cluster Ingress Controller request error rate high (default error
rate is 85%).
type: metric-prometheus
- enable: disable
expression: prom.ingress.ssl.expire
name: ingress-ssl-expire
notification:
message: "Cluster Ingress Controller SSL will expire in a few days (default
14 days). \nPromQL: ((nginx_ingress_controller_ssl_expire_time_seconds -
time()) / 24 / 3600) < 14"
type: metric-prometheus
- name: cluster-error
rules:
- enable: disable
expression: sls.app.ack.docker.hang
name: docker-hang
notification:
message: kubernetes node docker hang.
type: event
- enable: disable
expression: sls.app.ack.eviction
name: eviction-event
notification:
message: kubernetes eviction event.
type: event
- enable: disable
expression: sls.app.ack.gpu.xid_error
name: gpu-xid-error
notification:
message: kubernetes gpu xid error event.
type: event
- enable: disable
expression: sls.app.ack.image.pull_back_off
name: image-pull-back-off
notification:
message: kubernetes image pull back off event.
type: event
- enable: disable
expression: sls.app.ack.node.down
name: node-down
notification:
message: kubernetes node down event.
type: event
- enable: disable
expression: sls.app.ack.node.restart
name: node-restart
notification:
message: kubernetes node restart event.
type: event
- enable: disable
expression: sls.app.ack.ntp.down
name: node-ntp-down
notification:
message: kubernetes node ntp down.
type: event
- enable: disable
expression: sls.app.ack.node.pleg_error
name: node-pleg-error
notification:
message: kubernetes node pleg error event.
type: event
- enable: disable
expression: sls.app.ack.ps.hang
name: ps-hang
notification:
message: kubernetes ps hang event.
type: event
- enable: disable
expression: sls.app.ack.node.fd_pressure
name: node-fd-pressure
notification:
message: kubernetes node fd pressure event.
type: event
- enable: disable
expression: sls.app.ack.node.pid_pressure
name: node-pid-pressure
notification:
message: kubernetes node pid pressure event.
type: event
- enable: disable
expression: sls.app.ack.ccm.del_node_failed
name: node-del-err
notification:
message: kubernetes delete node failed.
type: event
- enable: disable
expression: sls.app.ack.ccm.add_node_failed
name: node-add-err
notification:
message: kubernetes add node failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.run_command_fail
name: nlc-run-cmd-err
notification:
message: kubernetes node pool nlc run command failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.empty_task_cmd
name: nlc-empty-cmd
notification:
message: kubernetes node pool nlc delete node failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.url_mode_unimpl
name: nlc-url-m-unimp
notification:
message: kubernetes nodde pool nlc delete node failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.op_not_found
name: nlc-opt-no-found
notification:
message: kubernetes node pool nlc delete node failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.destroy_node_fail
name: nlc-des-node-err
notification:
message: kubernetes node pool nlc destory node failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.drain_node_fail
name: nlc-drain-node-err
notification:
message: kubernetes node pool nlc drain node failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.restart_ecs_wait_fail
name: nlc-restart-ecs-wait
notification:
message: kubernetes node pool nlc restart ecs wait timeout.
type: event
- enable: disable
expression: sls.app.ack.nlc.restart_ecs_fail
name: nlc-restart-ecs-err
notification:
message: kubernetes node pool nlc restart ecs failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.reset_ecs_fail
name: nlc-reset-ecs-err
notification:
message: kubernetes node pool nlc reset ecs failed.
type: event
- enable: disable
expression: sls.app.ack.nlc.repair_fail
name: nlc-sel-repair-err
notification:
message: kubernetes node pool nlc self repair failed.
type: event
- name: res-exceptions
rules:
- enable: disable
expression: cms.host.cpu.utilization
name: node_cpu_util_high
notification:
message: kubernetes cluster node cpu utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.host.memory.utilization
name: node_mem_util_high
notification:
message: kubernetes cluster node memory utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.host.disk.utilization
name: node_disk_util_high
notification:
message: kubernetes cluster node disk utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.host.public.network.utilization
name: node_public_net_util_high
notification:
message: kubernetes cluster node public network utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.host.fs.inode.utilization
name: node_fs_inode_util_high
notification:
message: kubernetes cluster node file system inode utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.slb.qps.utilization
name: slb_qps_util_high
notification:
message: kubernetes cluster slb qps utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.slb.traffic.tx.utilization
name: slb_traff_tx_util_high
notification:
message: kubernetes cluster slb traffic utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.slb.max.connection.utilization
name: slb_max_con_util_high
notification:
message: kubernetes cluster max connection utilization too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: percent
value: "85"
type: metric-cms
- enable: disable
expression: cms.slb.drop.connection
name: slb_drop_con_high
notification:
message: kubernetes cluster drop connection count per second too high.
thresholds:
- key: CMS_ESCALATIONS_CRITICAL_Threshold
unit: count
value: "1"
type: metric-cms
- enable: disable
expression: sls.app.ack.node.disk_pressure
name: node-disk-pressure
notification:
message: kubernetes node disk pressure event.
type: event
- enable: disable
expression: sls.app.ack.resource.insufficient
name: node-res-insufficient
notification:
message: kubernetes node resource insufficient.
type: event
- enable: disable
expression: sls.app.ack.ip.not_enough
name: node-ip-pressure
notification:
message: kubernetes ip not enough event.
type: event
- enable: disable
expression: sls.app.ack.csi.no_enough_disk_space
name: disk_space_press
notification:
message: kubernetes csi not enough disk space.
type: event
- name: cluster-scale
rules:
- enable: disable
expression: sls.app.ack.autoscaler.scaleup_group
name: autoscaler-scaleup
notification:
message: kubernetes autoscaler scale up.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.scaledown
name: autoscaler-scaledown
notification:
message: kubernetes autoscaler scale down.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.scaleup_timeout
name: autoscaler-scaleup-timeout
notification:
message: kubernetes autoscaler scale up timeout.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.scaledown_empty
name: autoscaler-scaledown-empty
notification:
message: kubernetes autoscaler scale down empty node.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.scaleup_group_failed
name: autoscaler-up-group-failed
notification:
message: kubernetes autoscaler scale up failed.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.cluster_unhealthy
name: autoscaler-cluster-unhealthy
notification:
message: kubernetes autoscaler error, cluster not healthy.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.delete_started_timeout
name: autoscaler-del-started
notification:
message: kubernetes autoscaler delete node started long ago.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.delete_unregistered
name: autoscaler-del-unregistered
notification:
message: kubernetes autoscaler delete unregistered node.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.scaledown_failed
name: autoscaler-scale-down-failed
notification:
message: kubernetes autoscaler scale down failed.
type: event
- enable: disable
expression: sls.app.ack.autoscaler.instance_expired
name: autoscaler-instance-expired
notification:
message: kubernetes autoscaler scale down instance expired.
type: event
- name: workload-exceptions
rules:
- enable: disable
expression: prom.job.failed
name: job-failed
notification:
message: "Cluster Job failed. \nPromQL: kube_job_status_failed{job=\"_kube-state-metrics\"}
> 0"
type: metric-prometheus
- enable: disable
expression: prom.deployment.replicaError
name: deployment-rep-err
notification:
message: "Cluster Deployment replication status error. \nPromQL: kube_deployment_spec_replicas{job=\"_kube-state-metrics\"}
!= kube_deployment_status_replicas_available{job=\"_kube-state-metrics\"}"
type: metric-prometheus
- enable: disable
expression: prom.daemonset.scheduledError
name: daemonset-status-err
notification:
message: "Cluster Daemonset pod status or scheduled error. \nPromQL: ((100
- kube_daemonset_status_number_ready{} / kube_daemonset_status_desired_number_scheduled{}
* 100) or (kube_daemonset_status_desired_number_scheduled{} - kube_daemonset_status_current_number_scheduled{}))
> 0"
type: metric-prometheus
- enable: disable
expression: prom.daemonset.misscheduled
name: daemonset-misscheduled
notification:
message: "Cluster Daemonset misscheduled. \nPromQL: kube_daemonset_status_number_misscheduled{job=\"_kube-state-metrics\"}
\ > 0"
type: metric-prometheus
- name: pod-exceptions
rules:
- enable: disable
expression: sls.app.ack.pod.oom
name: pod-oom
notification:
message: kubernetes pod oom event.
type: event
- enable: disable
expression: sls.app.ack.pod.failed
name: pod-failed
notification:
message: kubernetes pod start failed event.
type: event
- enable: disable
expression: prom.pod.status.notHealthy
name: pod-status-err
notification:
message: 'Pod status exception. \nPromQL: min_over_time(sum by (namespace,
pod, phase) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed", job="_kube-state-metrics"})[${mins}m:1m])
> 0'
type: metric-prometheus
- enable: disable
expression: prom.pod.status.crashLooping
name: pod-crashloop
notification:
message: 'Pod status exception. \nPromQL: sum_over_time(increase(kube_pod_container_status_restarts_total{job="_kube-state-metrics"}[1m])[${mins}m:1m])
> 3'
type: metric-prometheus
- name: cluster-storage-err
rules:
- enable: disable
expression: sls.app.ack.csi.invalid_disk_size
name: csi_invalid_size
notification:
message: kubernetes csi invalid disk size.
type: event
- enable: disable
expression: sls.app.ack.csi.disk_not_portable
name: csi_not_portable
notification:
message: kubernetes csi not protable.
type: event
- enable: disable
expression: sls.app.ack.csi.deivce_busy
name: csi_device_busy
notification:
message: kubernetes csi disk device busy.
type: event
- enable: disable
expression: sls.app.ack.csi.no_ava_disk
name: csi_no_ava_disk
notification:
message: kubernetes csi no available disk.
type: event
- enable: disable
expression: sls.app.ack.csi.disk_iohang
name: csi_disk_iohang
notification:
message: kubernetes csi ioHang.
type: event
- enable: disable
expression: sls.app.ack.csi.latency_too_high
name: csi_latency_high
notification:
message: kubernetes csi pvc latency load too high.
type: event
- enable: disable
expression: prom.pv.failed
name: pv-failed
notification:
message: 'Cluster PersistentVolume failed. \nPromQL: kube_persistentvolume_status_phase{phase=~"Failed|Pending",
job="_kube-state-metrics"} > 0'
type: metric-prometheus
- name: cluster-network-err
rules:
- enable: disable
expression: sls.app.ack.ccm.no_ava_slb
name: slb-no-ava
notification:
message: kubernetes slb not available.
type: event
- enable: disable
expression: sls.app.ack.ccm.sync_slb_failed
name: slb-sync-err
notification:
message: kubernetes slb sync failed.
type: event
- enable: disable
expression: sls.app.ack.ccm.del_slb_failed
name: slb-del-err
notification:
message: kubernetes slb delete failed.
type: event
- enable: disable
expression: sls.app.ack.ccm.create_route_failed
name: route-create-err
notification:
message: kubernetes create route failed.
type: event
- enable: disable
expression: sls.app.ack.ccm.sync_route_failed
name: route-sync-err
notification:
message: kubernetes sync route failed.
type: event
- enable: disable
expression: sls.app.ack.terway.invalid_resource
name: terway-invalid-res
notification:
message: kubernetes terway have invalid resource.
type: event
- enable: disable
expression: sls.app.ack.terway.alloc_ip_fail
name: terway-alloc-ip-err
notification:
message: kubernetes terway allocate ip error.
type: event
- enable: disable
expression: sls.app.ack.terway.parse_fail
name: terway-parse-err
notification:
message: kubernetes terway parse k8s.aliyun.com/ingress-bandwidth annotation
error.
type: event
- enable: disable
expression: sls.app.ack.terway.allocate_failure
name: terway-alloc-res-err
notification:
message: kubernetes parse resource error.
type: event
- enable: disable
expression: sls.app.ack.terway.dispose_failure
name: terway-dispose-err
notification:
message: kubernetes dispose resource error.
type: event
- enable: disable
expression: sls.app.ack.terway.virtual_mode_change
name: terway-virt-mod-err
notification:
message: kubernetes virtual mode changed.
type: event
- enable: disable
expression: sls.app.ack.terway.config_check
name: terway-ip-check
notification:
message: kubernetes terway execute pod ip config check.
type: event
- enable: disable
expression: sls.app.ack.ingress.err_reload_nginx
name: ingress-reload-err
notification:
message: kubernetes ingress reload config error.
type: event
- name: security-err
rules:
- enable: disable
expression: sls.app.ack.si.config_audit_high_risk
name: si-c-a-risk
notification:
message: kubernetes high risks have be found after running config audit.
type: event
ruleVersion: v1.0.9
步驟四:分發報警規則到關聯集群中
報警規則實際也是一種Kubernetes資源。報警規則分發的原理和應用分發的原理一樣,都是通過開源Kubevela,將Fleet實例上的Kubernetes資源分發到關聯集群中。分發報警規則步驟如下:
使用以下模板創建分發規則ackalertrule-app.yaml。
方式一:將報警規則分發到打標production=true的關聯集群中。
執行以下命令,為關聯集群打標。
kubectl get managedclusters #獲取關聯集群clusterid。 kubectl label managedclusters <clusterid> production=true
將報警規則分發到打標production=true的關聯集群中。
apiVersion: core.oam.dev/v1beta1 kind: Application metadata: name: alertrules namespace: kube-system annotations: app.oam.dev/publishVersion: version1 spec: components: - name: alertrules type: ref-objects properties: objects: - resource: ackalertrules name: default policies: - type: topology name: prod-clusters properties: clusterSelector: production: "true" #通過標簽選擇集群。
方式二:可以直接輸入集群ID,將報警規則分發到指定的關聯集群。
替換以下
<clusterid>
為您需要下發的關聯集群的ID。apiVersion: core.oam.dev/v1beta1 kind: Application metadata: name: alertrules namespace: kube-system annotations: app.oam.dev/publishVersion: version1 spec: components: - name: alertrules type: ref-objects properties: objects: - resource: ackalertrules name: default policies: - type: topology name: prod-clusters properties: clusters: ["<clusterid1>", "<clusterid2>"] #通過clusterid選擇集群。
執行以下命令,創建分發規則。
kubectl apply -f ackalertrule-app.yaml
執行以下命令,查看分發執行狀態。
kubectl amc appstatus alertrules -n kube-system --tree --detail
預期輸出:
CLUSTER NAMESPACE RESOURCE STATUS APPLY_TIME DETAIL c565e4**** (cluster1)─── kube-system─── AckAlertRule/default updated 2022-**-** **:**:** Age: ** cbaa12**** (cluster2)─── kube-system─── AckAlertRule/default updated 2022-**-** **:**:** Age: **
修改報警規則
您可以通過以下步驟修改報警規則。
修改報警規則模板ackalertrule.yaml,并執行命令
kubectl apply -f ackalertrule.yaml
創建報警規則。修改分發模板ackalertrule-app.yaml,更新
annotations: app.oam.dev/publishVersion
,并執行kubectl apply -f ackalertrule-app.yaml
分發報警規則。