opsgenie_alertmanager

This example demonstrates how to configure Alertmanager to route alerts based on severity labels and environment. Alerts with severity "critical" and not in staging or development environments are sent to both Slack and Opsgenie. Other alerts are routed accordingly.

Alertmanager YAML Configuration Example

Below is a sample Alertmanager configuration in YAML format. This configuration defines how alerts are grouped, routed, and handled by different receivers such as Slack and Opsgenie.

# example showing how to route alerts based on severity labels
# if severity label is critical and environment is not -staging or -dev
# it routes to slack and opsgenie

global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/x/x/x'
  
route:
  group_by: ['alertname', 'cluster', 'job', 'env']
  repeat_interval: 24h
  group_interval: 5m
  # capture everything to default
  receiver: 'default'
  routes:
    # match and group severity=critical to the critical receiver
    # this will be sent to slack and opsgenie
    - match:
        severity: critical
      receiver: critical
      routes:
      # match then environment=*-staging to warning
      - match_re:
          environment: .*(-staging).*
        receiver: warning
      # match then environment=*-dev to default
      - match_re:
          environment: .*(-dev).*
        receiver: default

receivers:
  - name: 'default'
    slack_configs:
      - send_resolved: true
        title_link: 'http://alertmanager.homedns.xyz/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:confused:{{ else }}:sunglasses:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Description*: {{ .Annotations.description }}\n*Priority*: `{{ .Labels.severity | toUpper }}`\n{{ end }}"

  - name: 'warning'
    slack_configs:
      - send_resolved: true
        title_link: 'http://alertmanager.localdns.xyz/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:flushed:{{ else }}:sunglasses:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Description*: {{ .Annotations.description }}\n*Priority*: `{{ .Labels.severity | toUpper }}`\n{{ end }}"

  - name: 'critical'
    slack_configs:
      - send_resolved: true
        title_link: 'http://alertmanager.localdns.xyz/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:scream:{{ else }}:sunglasses:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Description*: {{ .Annotations.description }}\n*Priority*: `{{ .Labels.severity | toUpper }}`\n{{ end }}"
    opsgenie_configs:
      - send_resolved: true
        http_config: {}
        api_key: xx
        api_url: https://api.eu.opsgenie.com/
        message: '[{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        description: "{{ .CommonAnnotations.description }}\nMore Info:\n- Impact: {{ .CommonAnnotations.impact }}\n- Action: {{ .CommonAnnotations.action }}\n- Dashboard: {{ .CommonAnnotations.dashboard }}\n- Runbook: {{ .CommonAnnotations.runbook }}"
        source: '{{ template "opsgenie.default.source" . }}'
        responders:
        - name: DevOps
          type: team
        priority: '{{ if .CommonAnnotations.priority }}{{ .CommonAnnotations.priority }}{{ else }}P3{{ end }}'
        tags: 'monitoring, exporters, prometheus'
        note: 'test-note'

  - name: 'default-opsgenie'
    opsgenie_configs:
      - send_resolved: true
        http_config: {}
        api_key: xx
        api_url: https://api.eu.opsgenie.com/
        message: '{{ template "opsgenie.default.message" . }}'
        description: '{{ template "opsgenie.default.description" . }}'
        source: '{{ template "opsgenie.default.source" . }}'
        responders:
        - name: DevOps
          type: team
        priority: '{{ if .CommonAnnotations.priority }}{{ .CommonAnnotations.priority }}{{ else }}P3{{ end }}'
        tags: 'test-tag, devops'
        note: 'test-note'

This configuration uses YAML to define routing rules and receiver settings for Slack and Opsgenie. Ensure you replace placeholder values like API keys and URLs with your actual credentials.

Understanding Alert Routing

Alert routing in Alertmanager is determined by the route section. The routes subsection contains a list of route definitions. Each route defines a set of matching criteria (match or match_re) and a receiver. When an alert matches the criteria, it is sent to the specified receiver.

Configuring Receivers

Receivers define how alerts are handled. In this example, we have receivers for default, warning, and critical severities. Each receiver is configured to send alerts to Slack and/or Opsgenie. The slack_configs and opsgenie_configs sections contain the necessary configuration details for each service.

Further Resources

• Prometheus Alertmanager Documentation
• Opsgenie Documentation
• Slack Webhooks Documentation