kwatch has several controls for different kinds of noise. Choose the one that matches the problem you are trying to solve.

Group symptoms from one problem​

If many Pods fail because a node or shared dependency is unhealthy, investigate the common cause before creating separate response work for each Pod. kwatch correlates related observations into incidents and can group failures. The correlation settings explain cooldown and grouping windows.

Start with the defaults. Change a grouping threshold only after reviewing actual incidents: broad grouping can hide the distinction between independent failures.

Set a clear scope​

Namespace filters control what kwatch watches. If a cluster contains namespaces that your team does not own, configure scope before adding suppression rules. The general configuration guide explains allow and exclude lists, plus namespace label selection.

A scoped monitor still needs a route to the right team. Provider routes can filter by namespace, severity, or reason. For example, a production route can send urgent incidents to an on-call destination while another channel receives less urgent alerts. The channels guide shows how routing works.

Silence known, intentional failures​

Use a silence when a specific incident is expected and nobody should act on it, such as a planned maintenance state. Match narrowly by namespace, reason, or workload so the rule does not suppress unrelated incidents. A silence removes a matching finding from incident processing; it is stronger than sending an incident to a quieter channel.

Review each silence after maintenance ends. The silence reference lists the supported match fields.

Investigate missing alerts in order​

When you expected a message, do not immediately remove every noise control. Check the path in order:

1. Was kwatch's active leader ready and able to read the source?
2. Did the monitor detect the condition?
3. Did scope, baseline, a silence, cooldown, or grouping affect the decision?
4. Was an incident created but delivery failed?

This order separates a detection problem from an intentional suppression or a provider error. Use the missing-alert guide for the detailed checks.

Keep a response path​

The best noise rule leaves the next responder with enough context to act. Include a cluster name, route the incident to an owned channel, and attach a runbook where a known reason has a repeatable response. Test a provider change with kwatch lint --check when the provider supports a credential check, then verify a real notification through the supported diagnostic path.

This guide gives an operator a short path from symptom to evidence. kwatch can collect much of that context in an incident alert, but you should still verify the proposed cause against the workload.

1. Identify the failing container​

Start with the Pod name and namespace from the alert or workload status:

kubectl get pod -n production orders-api-7ffc9d4f9-x9p4t
kubectl describe pod -n production orders-api-7ffc9d4f9-x9p4t

In the description, look at the container's current and previous state, restart count, image, resource limits, probes, and recent Events. These details help separate an application exit from a failed health check or an image problem.

2. Read the previous container's logs​

A container in a restart loop may have already exited by the time you inspect it. Ask Kubernetes for the previous instance's logs:

kubectl logs -n production orders-api-7ffc9d4f9-x9p4t \
  --previous --container orders-api

Use the actual container name from kubectl describe pod. Logs can explain an application error or a missing environment variable, but they may be empty after a sudden termination. Kubernetes Events and the previous container state still matter.

3. Match the evidence to an action​

For example, an OOMKilled previous state means Kubernetes observed an out-of-memory termination. Compare the workload's memory limit with its usage and recent changes before raising the limit. Exit code 137 alone is not proof of an OOM kill: another SIGKILL can also produce 137.

A kwatch alert can bring the signal, affected workload, likely cause, recent logs, and Events together:

🚨 OOMKilled — production / orders-api
   Pod: orders-api-7ffc9d4f9-x9p4t
   Node: worker-3 · severity: high

💡 Cause: the container exceeded its 512Mi memory limit.
➡️ Next step: increase limits.memory or reduce memory usage.

This is an illustrative incident. Available evidence and wording depend on the failure and your configuration. Treat the next step as a starting point for investigation, not an automatic change to the workload.

4. Check scope and delivery if no alert arrived​

If kwatch did not notify you, confirm the active leader is ready, then check whether namespace filters, silence rules, startup baseline, cooldown, or grouping affected the incident. If an incident was created but delivery failed, check the provider configuration and delivery diagnostics. The missing-alert troubleshooting guide walks through this order.

Put the context where responders work​

The recommended installation path is the interactive kwatch manager. It selects a cluster, configures an alert destination, stores credentials in a Kubernetes Secret, and verifies the installation. The channel guides cover Slack, PagerDuty, and other destinations. For the full list of monitored Kubernetes signals, use the coverage guide.

The screenshots document the original notification flow. Use the current configuration guide for supported monitor settings.

Configure PVC monitoring​

The interactive manager is the supported installation and configuration path:

/bin/bash -c "$(curl -fsSL https://kwatch.dev/kwatch.sh)"

Choose Configure settings, then enable or tune the PVC monitor. The equivalent settings are:

pvcMonitor:
  enabled: true
  interval: 5
  threshold: 80
  criticalThreshold: 90
  clearThreshold: 75

threshold starts a warning, criticalThreshold raises severity for a more serious condition, and clearThreshold prevents a flapping alert from resolving too quickly. See the PVC monitor section for the full setting reference.

Choose an alert channel​

The manager can configure Teams, Slack, PagerDuty, email, webhooks, and other destinations. Every credential is stored in a Kubernetes Secret and referenced from the config file, for example:

alert:
  teams:
    webhook: "${file:/config/teams-webhook}"

Use the channel guides to choose a provider and the complete provider reference to see every field.

Verify the alert​

After the manager reports the workload as ready, check the PVC monitor metrics and wait for the next configured sweep. The example screenshots show the original Teams notification flow:

You can validate the configuration with kwatch lint. Keep thresholds below your storage provider's hard limit and pair PVC alerts with a runbook explaining how your team expands or cleans up the volume.

Thank you to the community​

Contributors improved alert destinations, documentation, and Kubernetes monitoring while operators tried kwatch in their clusters and shared feedback. See the contributors list for the people behind those changes.

The project has evolved since 2022. The current kwatch turns Kubernetes failures into incident alerts with a likely cause, supporting evidence, and a next step. It runs in your own cluster and can send alerts to the tools your team already uses.

Try the current release​

Use the getting-started guide for a quick tour, then install with the interactive manager. Historical release notes in this blog describe earlier versions and are not installation instructions for the current release.

If you have an idea or find a problem, open an issue or join the community.

🚀 What's New​

• Allow filtering by reasons by @simonfrey in #104
• feat: truncate logs to respect Discord's API requirements by @Tchoupinax in #105
• Fix copy-paste mistake for reasons parsing by @simonfrey in #110
• Run in a namespace context by @kdihalas in #115
• Fix typo by @unixfox in #95

🧰 Maintenance​

• Improve Documentation by @oguzhanoyan in #87
• Refactor/helm by @eremeevfd in #88
• ⬆️ Bump github.com/spf13/viper from 1.10.1 to 1.11.0 by @dependabot in #89
• ⬆️ Bump github.com/bwmarrin/discordgo from 0.24.0 to 0.25.0 by @dependabot in #90
• ⬆️ Bump k8s.io/client-go from 0.23.5 to 0.23.6 by @dependabot in #93
• ⬆️ Bump github.com/slack-go/slack from 0.10.2 to 0.10.3 by @dependabot in #94
• ⬆️ Bump k8s.io/client-go from 0.23.6 to 0.24.0 by @dependabot in #96
• ⬆️ Bump k8s.io/client-go from 0.24.0 to 0.24.1 by @dependabot in #99
• ⬆️ Bump github.com/spf13/viper from 1.11.0 to 1.12.0 by @dependabot in #102
• ⬆️ Bump github.com/slack-go/slack from 0.10.3 to 0.11.0 by @dependabot in #106
• ⬆️ Bump k8s.io/client-go from 0.24.1 to 0.24.2 by @dependabot in #107
• ⬆️ Bump k8s.io/client-go from 0.24.2 to 0.24.3 by @dependabot in #114
• ⬆️ Bump github.com/sirupsen/logrus from 1.8.1 to 1.9.0 by @dependabot in #116
• ⬆️ Bump github.com/slack-go/slack from 0.11.0 to 0.11.1 by @dependabot in #117

⬆️ How to Upgrade​

For a current installation, run the manager again and choose Upgrade. It handles release selection, readiness checks, and the Secret-backed configuration for you.

🌟 New Contributors​

• @oguzhanoyan made their first contribution in https://github.com/abahmed/kwatch/pull/87
• @eremeevfdmade their first contribution in https://github.com/abahmed/kwatch/pull/88
• @unixfox made their first contribution in https://github.com/abahmed/kwatch/pull/95
• @Tchoupinax made their first contribution in https://github.com/abahmed/kwatch/pull/105
• @kdihalas made their first contribution in https://github.com/abahmed/kwatch/pull/115

If you like kwatch, give it a star on GitHub!

🚀 What's New​

• Use slack blocks in slack handler by @simonfrey in https://github.com/abahmed/kwatch/pull/63
• Feature/allow to disable update check by @simonfrey in https://github.com/abahmed/kwatch/pull/67
• Add namespace forbid list, to allow all namespaces except certain ones by @simonfrey in https://github.com/abahmed/kwatch/pull/70

🧰 Maintenance​

• ⬆️ Bump github.com/slack-go/slack from 0.10.1 to 0.10.2 by @dependabot in https://github.com/abahmed/kwatch/pull/69
• add(Helm): by @yaskinny in https://github.com/abahmed/kwatch/pull/74
• ⬆️ Bump k8s.io/client-go from 0.23.3 to 0.23.4 by @dependabot in https://github.com/abahmed/kwatch/pull/73
• ⬆️ Bump github.com/bwmarrin/discordgo from 0.23.2 to 0.24.0 by @dependabot in https://github.com/abahmed/kwatch/pull/77
• ⬆️ Bump k8s.io/client-go from 0.23.4 to 0.23.5 by @dependabot in https://github.com/abahmed/kwatch/pull/80

⬆️ How to Upgrade​

For a current installation, run the manager again and choose Upgrade. It handles release selection, readiness checks, and the Secret-backed configuration for you.

🌟 New Contributors​

• @romualdofernandes made their first contribution in https://github.com/abahmed/kwatch/pull/45
• @dependabotmade their first contribution in https://github.com/abahmed/kwatch/pull/46
• @simonfrey made their first contribution in https://github.com/abahmed/kwatch/pull/54

If you like kwatch, give it a star on GitHub!

🚀 What's New​

• Support Mattermost for alerts by @abahmed in https://github.com/abahmed/kwatch/pull/53
• Support Opsgenie by @abahmed in https://github.com/abahmed/kwatch/pull/60

🐛 Bug Fixes​

• provider/telegram.go: add header content-type json by @romualdofernandes in https://github.com/abahmed/kwatch/pull/45
• Add PodInitializing as ignored reason by @simonfrey in https://github.com/abahmed/kwatch/pull/54
• Allow to ignore if a container has to forcefully shutdown due to graceful shutdown not working by @simonfrey in https://github.com/abahmed/kwatch/pull/58

🧰 Maintenance​

• build(deps): bump k8s.io/apimachinery from 0.23.1 to 0.23.2 by @dependabot in https://github.com/abahmed/kwatch/pull/46
• build(deps): bump k8s.io/api from 0.23.1 to 0.23.2 by @dependabot in https://github.com/abahmed/kwatch/pull/47
• build(deps): bump k8s.io/client-go from 0.23.1 to 0.23.2 by @dependabot in https://github.com/abahmed/kwatch/pull/48
• build(deps): bump k8s.io/client-go from 0.23.2 to 0.23.3 by @dependabot in https://github.com/abahmed/kwatch/pull/55

⬆️ How to Upgrade​

For a current installation, run the manager again and choose Upgrade. It selects the release, verifies the workload, and preserves the Secret-backed configuration. The old manifest-based workflow from this release is intentionally not used by the current supported workflow.

🌟 New Contributors​

• @romualdofernandes made their first contribution in https://github.com/abahmed/kwatch/pull/45
• @dependabotmade their first contribution in https://github.com/abahmed/kwatch/pull/46
• @simonfrey made their first contribution in https://github.com/abahmed/kwatch/pull/54

If you like kwatch, give it a star on GitHub!

The screenshots are from the original guide. Check the current PagerDuty channel guide before configuring a new integration.

1. Create a PagerDuty integration​

Create a service, choose an escalation policy, and add the Events API v2 integration. PagerDuty's escalation policy guide and Events API reference show the current UI and integration options.

Copy the service's integration key. It is a credential and must be stored in a Kubernetes Secret, never as a plain value in a ConfigMap or Git repository.

2. Install kwatch with kwatch.sh​

Run the supported interactive installer:

/bin/bash -c "$(curl -fsSL https://kwatch.dev/kwatch.sh)"

Choose Configure notification, select PagerDuty, and provide the integration key. The generated configuration uses a Secret-backed file:

alert:
  pagerduty:
    integrationKey: "${file:/config/pagerduty-integration-key}"

The manager installs the matching release and waits for readiness. It also handles future configuration changes and upgrades; do not apply old release manifests from this historical article.

3. Verify delivery​

Run kwatch lint --check where PagerDuty credential checks are supported, or use the diagnostics /test-alert endpoint for an end-to-end notification. The PagerDuty channel guide covers routing, retry, fallback, and compact delivery options.

You should now receive a PagerDuty incident when kwatch detects a configured Kubernetes failure:

For more destinations, browse the complete provider reference.

This guide shows how to connect Slack and install kwatch with the supported interactive manager. The manager keeps the webhook in a Kubernetes Secret; do not paste it directly into config.yaml.

The screenshots show the original Slack setup flow. Use the current Slack guide for supported settings and delivery options.

1. Create a Slack incoming webhook​

Create or choose the channel where you want incidents to arrive, then create an incoming webhook from the Slack API app settings. The screenshots below follow the Slack setup flow used by the original guide.

Copy the webhook URL. Treat it as a credential and keep it out of shell history, Git, and plain YAML files.

2. Install kwatch with kwatch.sh​

Run the manager from a machine with kubectl access to your cluster:

/bin/bash -c "$(curl -fsSL https://kwatch.dev/kwatch.sh)"

Choose Configure notification, select Slack, and paste the webhook when the manager asks for it. It writes a Secret-backed file reference such as:

alert:
  slack:
    webhook: "${file:/config/slack-webhook}"

The manager then installs the release, waits for the workload to become ready, and sends a startup notification. Run the interactive manager again whenever you want to inspect or manage the installation.

3. Test the notification​

Use kwatch lint --check for provider validation where supported. For a real message, enable healthCheck.diagnostics in the configuration and call the /test-alert endpoint. See the Slack channel guide for bot-token mode, routing, retry, and fallback options.

When a workload fails, kwatch sends the incident reason and available evidence to the configured channel:

Next steps​

Add routes if Slack should receive only production or high-severity incidents, or configure a fallback provider for delivery failures. The complete provider reference lists every supported alert field.

If you like kwatch, give it a star on GitHub and share feedback in Discord or the issue tracker.

Kubernetes gives teams a powerful way to run applications, but a failed pod or stalled rollout can leave you with a short status such as CrashLoopBackOff and not enough context to act quickly.

kwatch watches Kubernetes resources, events, and recent logs. When something needs attention, it groups related symptoms and sends an alert with the likely cause, impact, evidence, and a practical next step.

What kwatch watches​

• Pod crashes, restart loops, and containers that are not ready
• Nodes, scheduling delays, rollouts, StatefulSets, DaemonSets, Jobs, and CronJobs
• HPA pressure, PVC usage, service endpoints, ingress backends, and control-plane signals
• Optional TLS expiry checks, active probes, heartbeat notifications, and runtime telemetry

It runs as a small workload inside your cluster. You do not need Prometheus or a separate dashboard to receive the first useful alert.

What an alert contains​

Instead of only receiving OOMKilled, a notification can explain which workload was affected, include recent evidence, connect related symptoms, and point to the next thing to check. Alerts can be delivered to Slack, PagerDuty, Discord, email, webhooks, and dozens of other providers.

Try it​

The supported installation path is the interactive kwatch.sh manager. It creates the Kubernetes Secret, configures the selected provider, and installs the matching release resources:

/bin/bash -c "$(curl -fsSL https://kwatch.dev/kwatch.sh)"

Read the getting started guide for the shortest path to a working cluster, or explore the complete configuration reference when you want to tune monitors and alert routing.