HTTP Alerts
This document details the Prometheus rules for monitoring HTTP request rates and generating alerts based on specific criteria. The rules monitor for high 4xx response rates and excessive requests to the actuator endpoint, ensuring the health and performance of your HTTP services.
Understanding High 4xx Response Rate Alerts
This alert triggers when the rate of 4xx HTTP response codes exceeds 90% of total requests over a 1-minute window. This indicates potential client-side issues or API misuse.
Understanding High Actuator Rate Alerts
This alert triggers when the rate of requests to the actuator endpoint exceeds 60 requests per minute. Excessive actuator scrapes can impact performance and should be monitored.
Configuring Prometheus Alert Rules for HTTP Monitoring
Below is the configuration for Prometheus alert rules, designed for easy integration into your Alertmanager setup.
groups:
- name: http_alerts.yaml
rules:
- alert: high_4xx_response_rate
expr: sum(rate(http_server_requests_seconds_count{status=~"^4.."}[1m])) by (service) / sum(rate(http_server_requests_seconds_count[1m])) by (service) * 100 > 90
for: 1m
labels:
severity: warning
alert_channel: slack
team: devops
annotations:
title: "High Rate of 4xx Response Status {{ $labels.service }}"
description: "Too many HTTP requests with status 4xx (> 90%)\n VALUE = {{ humanize $value}}%\n SERVICE = {{ $labels.service }}"
summary: "High HTTP 4xx error rate (instance {{ $labels.instance }})"
- alert: high_actuator_rate
expr: sum(irate(http_server_requests_seconds_count{uri="/actuator/prometheus"}[1m])) > 60
for: 1m
labels:
severity: warning
alert_channel: slack
team: devops
annotations:
title: "High Rate of Actuator Scrapes {{ $labels.service }}"
description: "Too many HTTP requests against the actuator endpoint (> 1ps)\n VALUE = {{ $value}}\n SERVICE = {{ $labels.service }}"
summary: "High HTTP actuator endpoint rate (instance {{ $labels.instance }})"
Further Resources
For further information on Prometheus and Alertmanager, refer to the following resources: