This guide demonstrates how to create and utilize custom log metrics from Promtail, enabling you to extract valuable insights from your logs and expose them for Prometheus to scrape and analyze. By leveraging Promtail's pipeline stages, specifically the `metrics` stage, you can transform log lines into Prometheus-compatible metrics.

Setting Up Promtail for Custom Metrics

First, configure your Promtail agent to process logs and define custom metrics. This involves specifying the log source, labels, and the pipeline stages for metric extraction. Below is an example of a promtail-config.yml file:

$ cat /etc/promtail/promtail-config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: https://x:x@loki.domain.com/loki/api/v1/push

scrape_configs:
  - job_name: prod/logs
    static_configs:
    - targets:
        - localhost
      labels:
        job: prod/logs
        service: nginx-test
        logtype: info
        __path__: /var/log/nginx/access.log
    pipeline_stages:
    - match:
        selector: '{job="prod/logs", service="nginx-test"}'
        stages:
        - regex:
            expression: '.*(?P<hits>GET /.*)'
        - metrics:
            nginx_get_hits:
              type: Counter
              description: "Total GET requests"
              source: hits
              config:
                action: inc

Verifying Metrics Endpoint

After configuring Promtail, verify that the metrics are being exposed correctly by accessing the /metrics endpoint. This endpoint provides the metrics in Prometheus exposition format.

$ curl http://localhost:9080/metrics
# HELP promtail_custom_nginx_get_hits Total GET requests
# TYPE promtail_custom_nginx_get_hits counter
promtail_custom_nginx_get_hits{filename="/var/log/nginx/access.log",job="prod/logs",logtype="info",service="nginx-test"} 801

Configuring Prometheus to Scrape Promtail Metrics

Next, configure Prometheus to scrape the metrics exposed by Promtail. This example uses ec2_sd_configs for service discovery, assuming your Promtail instances are running on EC2 instances tagged appropriately.

scrape_configs:
  # promtail-exporter
  - job_name: promtail-exporter
    scrape_interval: 15s
    ec2_sd_configs:
    - region: eu-west-1
      port: 9100
      filters:
        - name: tag:PromtailScrape
          values:
            - Enabled
    relabel_configs:
    - source_labels: [__meta_ec2_private_ip]
      replacement: '${1}:9080'
      target_label: __address__
    - source_labels: [__meta_ec2_tag_Name]
      target_label: instance

Ensure your EC2 instances are tagged with PromtailScrape=Enabled for Prometheus to discover and scrape them. You can verify this in Prometheus by querying up{job="promtail-exporter"}.

Querying Custom Metrics with PromQL

Once Prometheus is scraping the metrics, you can query them using PromQL. For instance, to see the increase in GET requests over a 5-minute interval:

increase(promtail_custom_nginx_get_hits{service="nginx-test"}[5m])

Setting Up Alerts Based on Custom Metrics

Custom log metrics are invaluable for setting up alerts. You can define alerting rules in Prometheus to notify you when certain thresholds are breached. For example, to alert if the number of GET requests exceeds 5000 within a 5-minute window:

  - name: loki-metric-alert
    groups:
      - name: loki_metric_alert
        rules:
        - alert: nginx_get_hits
          expr: sum(increase(promtail_custom_nginx_get_hits{service="nginx-test"}[5m])) > 5000
          for: 2m

Further Exploration

For more advanced metric configurations and troubleshooting, refer to the official Promtail documentation on pipeline stages and Prometheus documentation on PromQL functions.