Skip to content

Add resorucequota related monitoring alerts#1057

Open
w13915984028 wants to merge 1 commit into
harvester:mainfrom
w13915984028:quota-alert
Open

Add resorucequota related monitoring alerts#1057
w13915984028 wants to merge 1 commit into
harvester:mainfrom
w13915984028:quota-alert

Conversation

@w13915984028

@w13915984028 w13915984028 commented Jun 3, 2026

Copy link
Copy Markdown
Member

Problem:

How to observe the monitoring and alerts upon resourcequota.

Solution:

Explain why and how to observe resourcequota related alerts.

Related Issue(s):

harvester/harvester#10093
harvester/harvester#10267
#981

Test plan:

Additional documentation or context

A simple webhook server to receive the alerts from Harvester cluster, it got below alert, with key information:

'namespace': 'quota-test2'
'resourcequota': 'default-5k2gw'
'description': 'Namespace quota-test2 is using 91.5% of its limits.memory quota on cluster .`

{'receiver': 'quota-test2/amc-example2/amc-webhook-receiver', 'status': 'firing', 'alerts': [{'status': 'firing', 'labels': {'alertname': 'KubeQuotaAlmostFull', 'container': 'kube-state-metrics', 'endpoint': 'http', 'namespace': 'quota-test2', 'pod': 'rancher-monitoring-kube-state-metrics-9585db8c8-2kbbq', 'prometheus': 'cattle-monitoring-system/rancher-monitoring-prometheus', 'resource': 'limits.memory', 'resourcequota': 'default-5k2gw', 'service': 'rancher-monitoring-kube-state-metrics', 'severity': 'info'}, 'annotations': {'description': 'Namespace quota-test2 is using 91.5% of its limits.memory quota on cluster .', 'runbook_url': 'https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull', 'summary': 'Namespace quota is going to be full.'}, 'startsAt': '2026-06-03T16:23:00.904Z', 'endsAt': '0001-01-01T00:00:00Z', 'generatorURL': 'https://192.168.122.141/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-prometheus:9090/proxy/graph?g0.expr=kube_resourcequota%7Bjob%3D%22kube-state-metrics%22%2Ctype%3D%22used%22%7D+%2F+ignoring+%28instance%2C+job%2C+type%29+%28kube_resourcequota%7Bjob%3D%22kube-state-metrics%22%2Ctype%3D%22hard%22%7D+%3E+0%29+%3E+0.9+%3C+1&g0.tab=1', 'fingerprint': '686d97b5cf5e722f'}], 'groupLabels': {'alertname': 'KubeQuotaAlmostFull'}, 'commonLabels': {'alertname': 'KubeQuotaAlmostFull', 'container': 'kube-state-metrics', 'endpoint': 'http', 'namespace': 'quota-test2', 'pod': 'rancher-monitoring-kube-state-metrics-9585db8c8-2kbbq', 'prometheus': 'cattle-monitoring-system/rancher-monitoring-prometheus', 'resource': 'limits.memory', 'resourcequota': 'default-5k2gw', 'service': 'rancher-monitoring-kube-state-metrics', 'severity': 'info'}, 'commonAnnotations': {'description': 'Namespace quota-test2 is using 91.5% of its limits.memory quota on cluster .', 'runbook_url': 'https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubequotaalmostfull', 'summary': 'Namespace quota is going to be full.'}, 'externalURL': 'https://192.168.122.141/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-alertmanager:9093/proxy', 'version': '4', 'groupKey': '{}/{namespace="quota-test2"}:{alertname="KubeQuotaAlmostFull"}', 'truncatedAlerts': 0}

Copilot AI review requested due to automatic review settings June 3, 2026 19:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Resource Quotas documentation to explain how to observe ResourceQuota monitoring/alerts when the rancher-monitoring addon is enabled.

Changes:

  • Adds a new section describing pre-configured ResourceQuota-related Prometheus alert rules.
  • Adds step-by-step instructions for viewing ResourceQuota alerts in the Harvester UI / Prometheus.
  • Adds an example kubectl output snippet and a screenshot reference for alert status.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/rancher/resource-quota.md Outdated
Comment thread docs/rancher/resource-quota.md Outdated
Comment thread docs/rancher/resource-quota.md
@w13915984028 w13915984028 force-pushed the quota-alert branch 2 times, most recently from f78f361 to 6d82ae0 Compare June 3, 2026 19:28
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Name Link
🔨 Latest commit a6645b3
😎 Deploy Preview https://6a208a633ab7478dea5b4a3e--harvester-preview.netlify.app

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated 4 comments.

Comment thread docs/rancher/resource-quota.md Outdated
Comment thread docs/rancher/resource-quota.md
Comment thread docs/rancher/resource-quota.md Outdated
Comment thread docs/rancher/resource-quota.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated 2 comments.

Comment thread docs/rancher/resource-quota.md Outdated
Comment thread docs/rancher/resource-quota.md Outdated
Signed-off-by: Jian Wang <jian.wang@suse.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants