{ "version": "https://jsonfeed.org/version/1", "title": "kwatch Blog", "home_page_url": "https://kwatch.dev/blog", "description": "Kubernetes incident response guides and kwatch project history", "items": [ { "id": "https://kwatch.dev/blog/reduce-kubernetes-alert-noise", "content_html": "

An alert is useful when it changes what the on-call person does. Repeated\nnotifications for one failure, or dozens of symptoms from one shared problem,\nmake that decision harder. Muting everything is risky too: a real incident can\ndisappear with the noise.

\n

kwatch has several controls for different kinds of noise. Choose the one that\nmatches the problem you are trying to solve.

\n

Group symptoms from one problem​

\n

If many Pods fail because a node or shared dependency is unhealthy, investigate\nthe common cause before creating separate response work for each Pod. kwatch\ncorrelates related observations into incidents and can group failures. The\ncorrelation settings\nexplain cooldown and grouping windows.

\n

Start with the defaults. Change a grouping threshold only after reviewing\nactual incidents: broad grouping can hide the distinction between independent\nfailures.

\n

Set a clear scope​

\n

Namespace filters control what kwatch watches. If a cluster contains\nnamespaces that your team does not own, configure scope before adding\nsuppression rules. The general configuration guide\nexplains allow and exclude lists, plus namespace label selection.

\n

A scoped monitor still needs a route to the right team. Provider routes can\nfilter by namespace, severity, or reason. For example, a production route can\nsend urgent incidents to an on-call destination while another channel receives\nless urgent alerts. The channels guide shows how routing works.

\n

Silence known, intentional failures​

\n

Use a silence when a specific incident is expected and nobody should act on\nit, such as a planned maintenance state. Match narrowly by namespace, reason,\nor workload so the rule does not suppress unrelated incidents. A silence\nremoves a matching finding from incident processing; it is stronger than\nsending an incident to a quieter channel.

\n

Review each silence after maintenance ends. The\nsilence reference\nlists the supported match fields.

\n

Investigate missing alerts in order​

\n

When you expected a message, do not immediately remove every noise control.\nCheck the path in order:

\n
    \n
  1. Was kwatch's active leader ready and able to read the source?
    2. \n
  2. Did the monitor detect the condition?
    3. \n
  3. Did scope, baseline, a silence, cooldown, or grouping affect the decision?
    4. \n
  4. Was an incident created but delivery failed?
    5. \n
\n

This order separates a detection problem from an intentional suppression or a\nprovider error. Use the missing-alert guide\nfor the detailed checks.

\n

Keep a response path​

\n

The best noise rule leaves the next responder with enough context to act.\nInclude a cluster name, route the incident to an owned channel, and attach a\nrunbook where a known reason has a repeatable response. Test a provider change\nwith kwatch lint --check when the provider supports a credential check, then\nverify a real notification through the supported diagnostic path.

", "url": "https://kwatch.dev/blog/reduce-kubernetes-alert-noise", "title": "Reduce Kubernetes alert noise without hiding incidents", "summary": "Use kwatch grouping, cooldown, silences, namespace scope, and alert routes deliberately so responders see the incidents that need action.", "date_modified": "2026-09-28T00:00:00.000Z", "author": { "name": "kwatch contributors", "url": "https://github.com/abahmed/kwatch" }, "tags": [ "kwatch", "kubernetes", "alerting", "incident-response" ] }, { "id": "https://kwatch.dev/blog/troubleshoot-kubernetes-crashloopbackoff", "content_html": "

A Pod in CrashLoopBackOff is restarting after a container exits repeatedly.\nThe status tells you that a restart loop exists, but it does not identify one\nuniversal cause. Memory limits, application errors, missing configuration, and\nfailed dependencies need different fixes.

\n

This guide gives an operator a short path from symptom to evidence. kwatch can\ncollect much of that context in an incident alert, but you should still verify\nthe proposed cause against the workload.

\n

1. Identify the failing container​

\n

Start with the Pod name and namespace from the alert or workload status:

\n
kubectl get pod -n production orders-api-7ffc9d4f9-x9p4t
kubectl describe pod -n production orders-api-7ffc9d4f9-x9p4t
\n

In the description, look at the container's current and previous state, restart\ncount, image, resource limits, probes, and recent Events. These details help\nseparate an application exit from a failed health check or an image problem.

\n

2. Read the previous container's logs​

\n

A container in a restart loop may have already exited by the time you inspect\nit. Ask Kubernetes for the previous instance's logs:

\n
kubectl logs -n production orders-api-7ffc9d4f9-x9p4t \\
  --previous --container orders-api
\n

Use the actual container name from kubectl describe pod. Logs can explain an\napplication error or a missing environment variable, but they may be empty\nafter a sudden termination. Kubernetes Events and the previous container state\nstill matter.

\n

3. Match the evidence to an action​

\n

For example, an OOMKilled previous state means Kubernetes observed an\nout-of-memory termination. Compare the workload's memory limit with its usage\nand recent changes before raising the limit. Exit code 137 alone is not proof\nof an OOM kill: another SIGKILL can also produce 137.

\n

A kwatch alert can bring the signal, affected workload, likely cause, recent\nlogs, and Events together:

\n
🚨 OOMKilled — production / orders-api
   Pod: orders-api-7ffc9d4f9-x9p4t
   Node: worker-3 · severity: high

💡 Cause: the container exceeded its 512Mi memory limit.
➡️ Next step: increase limits.memory or reduce memory usage.
\n

This is an illustrative incident. Available evidence and wording depend on the\nfailure and your configuration. Treat the next step as a starting point for\ninvestigation, not an automatic change to the workload.

\n

4. Check scope and delivery if no alert arrived​

\n

If kwatch did not notify you, confirm the active leader is ready, then check\nwhether namespace filters, silence rules, startup baseline, cooldown, or\ngrouping affected the incident. If an incident was created but delivery failed,\ncheck the provider configuration and delivery diagnostics. The\nmissing-alert troubleshooting guide walks\nthrough this order.

\n

Put the context where responders work​

\n

The recommended installation path is the interactive\nkwatch manager. It selects a cluster, configures an alert\ndestination, stores credentials in a Kubernetes Secret, and verifies the\ninstallation. The channel guides cover Slack, PagerDuty, and\nother destinations. For the full list of monitored Kubernetes signals, use the\ncoverage guide.

", "url": "https://kwatch.dev/blog/troubleshoot-kubernetes-crashloopbackoff", "title": "Troubleshoot Kubernetes CrashLoopBackOff with incident context", "summary": "A practical CrashLoopBackOff triage path: inspect the previous container state, logs, events, and workload changes, then use kwatch to send the context in one alert.", "date_modified": "2026-09-28T00:00:00.000Z", "author": { "name": "kwatch contributors", "url": "https://github.com/abahmed/kwatch" }, "tags": [ "kwatch", "kubernetes", "monitoring", "troubleshooting" ] }, { "id": "https://kwatch.dev/blog/monitor-pvc-usage", "content_html": "

A full PersistentVolumeClaim can take an application down even when every pod\nlooks healthy. kwatch's PVC monitor watches usage and sends a warning or critical\nalert before the volume reaches capacity.

\n
\n

The screenshots document the original notification flow. Use the current\nconfiguration guide for supported monitor\nsettings.

\n
\n

Configure PVC monitoring​

\n

The interactive manager is the supported installation and configuration path:

\n
/bin/bash -c \"$(curl -fsSL https://kwatch.dev/kwatch.sh)\"
\n

Choose Configure settings, then enable or tune the PVC monitor. The\nequivalent settings are:

\n
pvcMonitor:
  enabled: true
  interval: 5
  threshold: 80
  criticalThreshold: 90
  clearThreshold: 75
\n

threshold starts a warning, criticalThreshold raises severity for a more\nserious condition, and clearThreshold prevents a flapping alert from\nresolving too quickly. See the PVC monitor section\nfor the full setting reference.

\n

Choose an alert channel​

\n

The manager can configure Teams, Slack, PagerDuty, email, webhooks, and other\ndestinations. Every credential is stored in a Kubernetes Secret and referenced\nfrom the config file, for example:

\n
alert:
  teams:
    webhook: \"${file:/config/teams-webhook}\"
\n

Use the channel guides to choose a provider and the\ncomplete provider reference to see every field.

\n

Verify the alert​

\n

After the manager reports the workload as ready, check the PVC monitor metrics\nand wait for the next configured sweep. The example screenshots show the\noriginal Teams notification flow:

\n

\"Configure

\n

\"kwatch

\n

You can validate the configuration with kwatch lint. Keep thresholds below\nyour storage provider's hard limit and pair PVC alerts with a runbook explaining\nhow your team expands or cleans up the volume.

", "url": "https://kwatch.dev/blog/monitor-pvc-usage", "title": "Monitor Kubernetes PVC usage with kwatch", "summary": "Monitor Kubernetes PersistentVolumeClaims with kwatch and receive alerts before storage pressure becomes an incident.", "date_modified": "2023-03-14T00:00:00.000Z", "author": { "name": "Abdelrahman Ahmed", "url": "https://github.com/abahmed" }, "tags": [ "kwatch", "kubernetes", "monitoring", "alerting", "pvc" ] }, { "id": "https://kwatch.dev/blog/kwatch-hits-100k-downloads", "content_html": "

In July 2022, the kwatch container package\npassed 100,000 pulls. This post records that historical milestone and thanks\nthe people who helped the project reach it.

\n

Thank you to the community​

\n

Contributors improved alert destinations, documentation, and Kubernetes\nmonitoring while operators tried kwatch in their clusters and shared feedback.\nSee the contributors list\nfor the people behind those changes.

\n

The project has evolved since 2022. The current kwatch turns Kubernetes\nfailures into incident alerts with a likely cause, supporting evidence, and a\nnext step. It runs in your own cluster and can send alerts to the tools your\nteam already uses.

\n

Try the current release​

\n

Use the getting-started guide for a quick tour, then install with the\ninteractive manager. Historical release notes in this\nblog describe earlier versions and are not installation instructions for the\ncurrent release.

\n

If you have an idea or find a problem, open an\nissue or join the\ncommunity.

", "url": "https://kwatch.dev/blog/kwatch-hits-100k-downloads", "title": "kwatch reached 100,000 container pulls in 2022", "summary": "A July 2022 kwatch project milestone: 100,000 container pulls, thanks to contributors and the community. Find the current installation guide here.", "date_modified": "2022-07-24T00:00:00.000Z", "author": { "name": "Abdelrahman Ahmed", "url": "https://github.com/abahmed" }, "tags": [ "kwatch", "kubernetes", "docker", "downloads", "100k" ] }, { "id": "https://kwatch.dev/blog/release-v0-6-0", "content_html": "

These are the historical notes for kwatch v0.6.0, released in 2022. They record the changes and contributors at that point in the project. For a current installation or upgrade, use the interactive manager.

\n

🚀 What's New​

\n\n

🧰 Maintenance​

\n\n

⬆️ How to Upgrade​

\n

For a current installation, run the manager again and choose Upgrade. It\nhandles release selection, readiness checks, and the Secret-backed configuration\nfor you.

\n

🌟 New Contributors​

\n\n

If you like kwatch, give it a star on GitHub!

", "url": "https://kwatch.dev/blog/release-v0-6-0", "title": "kwatch v0.6.0 release notes (2022)", "summary": "Historical kwatch v0.6.0 release notes, contributors, and changes from 2022. Use the current installer guide for supported installation and upgrades.", "date_modified": "2022-07-24T00:00:00.000Z", "author": { "name": "Abdelrahman Ahmed", "url": "https://github.com/abahmed" }, "tags": [ "kwatch", "kubernetes", "devops", "monitoring", "release" ] }, { "id": "https://kwatch.dev/blog/release-v0-5-0", "content_html": "

These are the historical notes for kwatch v0.5.0, released in 2022. They record the changes and contributors at that point in the project. For a current installation or upgrade, use the interactive manager.

\n

🚀 What's New​

\n\n

🧰 Maintenance​

\n\n

⬆️ How to Upgrade​

\n

For a current installation, run the manager again and choose Upgrade. It\nhandles release selection, readiness checks, and the Secret-backed configuration\nfor you.

\n

🌟 New Contributors​

\n\n

If you like kwatch, give it a star on GitHub!

", "url": "https://kwatch.dev/blog/release-v0-5-0", "title": "kwatch v0.5.0 release notes (2022)", "summary": "Historical kwatch v0.5.0 release notes, contributors, and changes from 2022. Use the current installer guide for supported installation and upgrades.", "date_modified": "2022-03-19T00:00:00.000Z", "author": { "name": "Abdelrahman Ahmed", "url": "https://github.com/abahmed" }, "tags": [ "kwatch", "kubernetes", "devops", "monitoring", "release" ] }, { "id": "https://kwatch.dev/blog/release-v0-4-0", "content_html": "

These are the historical notes for kwatch v0.4.0, released in 2022. They record the changes and contributors at that point in the project. For a current installation or upgrade, use the interactive manager.

\n

🚀 What's New​

\n\n

🐛 Bug Fixes​

\n\n

🧰 Maintenance​

\n\n

⬆️ How to Upgrade​

\n

For a current installation, run the manager again and choose Upgrade. It\nselects the release, verifies the workload, and preserves the Secret-backed\nconfiguration. The old manifest-based workflow from this release is\nintentionally not used by the current supported workflow.

\n

🌟 New Contributors​

\n\n

If you like kwatch, give it a star on GitHub!

", "url": "https://kwatch.dev/blog/release-v0-4-0", "title": "kwatch v0.4.0 release notes (2022)", "summary": "Historical kwatch v0.4.0 release notes, contributors, and changes from 2022. Use the current installer guide for supported installation and upgrades.", "date_modified": "2022-02-02T00:00:00.000Z", "author": { "name": "Abdelrahman Ahmed", "url": "https://github.com/abahmed" }, "tags": [ "kwatch", "kubernetes", "devops", "monitoring", "release" ] }, { "id": "https://kwatch.dev/blog/detect-crashes-in-your-cluster-using-kwatch-and-pagerduty", "content_html": "

PagerDuty is a good destination when a Kubernetes failure needs an on-call\nresponse. kwatch adds the Kubernetes context before it sends the incident: the\naffected workload, likely cause, recent evidence, and what to investigate next.

\n
\n

The screenshots are from the original guide. Check the current\nPagerDuty channel guide before configuring a\nnew integration.

\n
\n

1. Create a PagerDuty integration​

\n

Create a service, choose an escalation policy, and add the Events API v2\nintegration. PagerDuty's escalation policy guide\nand Events API reference\nshow the current UI and integration options.

\n

\"Create\n\"Configure\n\"Choose\n\"Add

\n

Copy the service's integration key. It is a credential and must be stored\nin a Kubernetes Secret, never as a plain value in a ConfigMap or Git repository.

\n

2. Install kwatch with kwatch.sh​

\n

Run the supported interactive installer:

\n
/bin/bash -c \"$(curl -fsSL https://kwatch.dev/kwatch.sh)\"
\n

Choose Configure notification, select PagerDuty, and provide the\nintegration key. The generated configuration uses a Secret-backed file:

\n
alert:
  pagerduty:
    integrationKey: \"${file:/config/pagerduty-integration-key}\"
\n

The manager installs the matching release and waits for readiness. It also\nhandles future configuration changes and upgrades; do not apply old release\nmanifests from this historical article.

\n

3. Verify delivery​

\n

Run kwatch lint --check where PagerDuty credential checks are supported, or\nuse the diagnostics /test-alert endpoint for an end-to-end notification.\nThe PagerDuty channel guide covers routing, retry,\nfallback, and compact delivery options.

\n

\"Copy

\n

You should now receive a PagerDuty incident when kwatch detects a configured\nKubernetes failure:

\n

\"PagerDuty

\n

For more destinations, browse the complete provider reference.

", "url": "https://kwatch.dev/blog/detect-crashes-in-your-cluster-using-kwatch-and-pagerduty", "title": "Send Kubernetes incident alerts to PagerDuty with kwatch", "summary": "Connect kwatch to PagerDuty so Kubernetes incidents arrive with a clear reason, evidence, and next step.", "date_modified": "2022-01-22T00:00:00.000Z", "author": { "name": "Amgad Ramses", "url": "https://github.com/amgadramses" }, "tags": [ "kwatch", "kubernetes", "monitoring", "alerting", "pagerduty" ] }, { "id": "https://kwatch.dev/blog/detect-crashes-in-your-cluster-using-kwatch-and-slack", "content_html": "

When a Kubernetes workload restarts, a Slack notification is useful only if it\nhelps you decide what to do next. kwatch turns crash signals, events, and recent\nlogs into a focused incident message.

\n

This guide shows how to connect Slack and install kwatch with the supported\ninteractive manager. The manager keeps the webhook in a Kubernetes Secret; do\nnot paste it directly into config.yaml.

\n
\n

The screenshots show the original Slack setup flow. Use the current\nSlack guide for supported settings and delivery\noptions.

\n
\n

1. Create a Slack incoming webhook​

\n

Create or choose the channel where you want incidents to arrive, then create an\nincoming webhook from the Slack API app settings.\nThe screenshots below follow the Slack setup flow used by the original guide.

\n

\"Create

\n

\"Create

\n

\"Enable

\n

\"Choose

\n

Copy the webhook URL. Treat it as a credential and keep it out of shell history,\nGit, and plain YAML files.

\n

2. Install kwatch with kwatch.sh​

\n

Run the manager from a machine with kubectl access to your cluster:

\n
/bin/bash -c \"$(curl -fsSL https://kwatch.dev/kwatch.sh)\"
\n

Choose Configure notification, select Slack, and paste the webhook when the\nmanager asks for it. It writes a Secret-backed file reference such as:

\n
alert:
  slack:
    webhook: \"${file:/config/slack-webhook}\"
\n

The manager then installs the release, waits for the workload to become ready,\nand sends a startup notification. Run the interactive manager again whenever\nyou want to inspect or manage the installation.

\n

3. Test the notification​

\n

Use kwatch lint --check for provider validation where supported. For a real\nmessage, enable healthCheck.diagnostics in the configuration and call the\n/test-alert endpoint. See the Slack channel guide for\nbot-token mode, routing, retry, and fallback options.

\n

\"kwatch

\n

When a workload fails, kwatch sends the incident reason and available evidence\nto the configured channel:

\n

\"kwatch

\n

Next steps​

\n

Add routes if Slack should receive only production or high-severity incidents,\nor configure a fallback provider for delivery failures. The complete provider\nreference lists every supported alert field.

\n

If you like kwatch, give it a star on GitHub\nand share feedback in Discord or the\nissue tracker.

", "url": "https://kwatch.dev/blog/detect-crashes-in-your-cluster-using-kwatch-and-slack", "title": "Detect Kubernetes crashes with kwatch and Slack", "summary": "Set up kwatch with Slack to receive clear Kubernetes incident alerts with the cause, evidence, and next step.", "date_modified": "2022-01-10T00:00:00.000Z", "author": { "name": "Andrew Attallah", "url": "https://github.com/Andrew44Ashraf" }, "tags": [ "kwatch", "kubernetes", "monitoring", "alerting", "slack" ] }, { "id": "https://kwatch.dev/blog/what-is-kwatch", "content_html": "

See what broke. Understand why. Know what to do next. 👀🧠⚡

\n

Kubernetes gives teams a powerful way to run applications, but a failed pod or\nstalled rollout can leave you with a short status such as CrashLoopBackOff and\nnot enough context to act quickly.

\n

kwatch watches Kubernetes resources, events, and recent logs. When something\nneeds attention, it groups related symptoms and sends an alert with the likely\ncause, impact, evidence, and a practical next step.

\n

What kwatch watches​

\n\n

It runs as a small workload inside your cluster. You do not need Prometheus or\na separate dashboard to receive the first useful alert.

\n

What an alert contains​

\n

Instead of only receiving OOMKilled, a notification can explain which\nworkload was affected, include recent evidence, connect related symptoms, and\npoint to the next thing to check. Alerts can be delivered to Slack, PagerDuty,\nDiscord, email, webhooks, and dozens of other providers.

\n

Try it​

\n

The supported installation path is the interactive kwatch.sh\nmanager. It creates the Kubernetes Secret, configures the selected provider,\nand installs the matching release resources:

\n
/bin/bash -c \"$(curl -fsSL https://kwatch.dev/kwatch.sh)\"
\n

Read the getting started guide for the shortest path to a working\ncluster, or explore the complete configuration reference\nwhen you want to tune monitors and alert routing.

", "url": "https://kwatch.dev/blog/what-is-kwatch", "title": "What is kwatch? Kubernetes incidents, explained", "summary": "See what broke, understand why, and know what to do next with open-source Kubernetes incident monitoring from kwatch.", "date_modified": "2022-01-05T00:00:00.000Z", "author": { "name": "Abdelrahman Ahmed", "url": "https://github.com/abahmed" }, "tags": [ "kwatch", "kubernetes", "monitoring", "alerting" ] } ] }