Prometheus Host Alert Rules

These Prometheus alert rules are designed to monitor the status of your nodes. An alert is triggered if a node becomes unreachable for more than one minute, ensuring timely intervention and minimizing potential downtime.

Understanding Prometheus Alerting

Prometheus is a powerful monitoring and alerting toolkit. These alert rules leverage Prometheus's capabilities to provide real-time insights into the health and availability of your infrastructure.

Alert Rules Configuration

Below is the configuration for the host alert rules. This YAML configuration defines the conditions under which alerts are triggered, as well as the associated labels and annotations that provide context and guidance for resolving the issue.

groups:
- name: host_alert_rules.yml
  rules:

  # Alert for any node that is unreachable for > 1 minute.
  - alert: node_down
    expr: up{job="node-exporter"} == 0
    for: 1m
    labels:
      severity: critical
      environment: foobar-production
    annotations:
      summary: "Job {{ $labels.job }} is down on {{ $labels.instance }}"
      description: "Failed to scrape {{ $labels.job }} on {{ $labels.instance }} for more than 1 minute. Node might be down."
      impact: "Any metrics from {{ $labels.job }} on {{ $labels.instance }} will be missing"
      action: "Check on {{ $labels.instance }} if {{ $labels.job }} is running"
      dashboard: https://grafana.localdns.xyz/d/pdfrTcGnQ/host-metrics
      runbook: https://mydocs.localdns.xyz/wiki/runbooks/1
      priority: P2

Analyzing the Alert Logic

The node_down alert is triggered when the up metric, collected by the node-exporter job, reports a value of 0. This indicates that the node is not responding. The for: 1m parameter ensures that the alert is only triggered after the node has been down for a full minute, preventing false positives due to transient network issues.

opsgenie_host_alert_rules

Understanding Prometheus Alerting

Alert Rules Configuration

Analyzing the Alert Logic

Further Reading