Promtail Relabeling: Stdout to Info Logs

This document provides a configuration example for Promtail to relabel log levels. Specifically, it demonstrates how to capture standard output (stdout) and standard error (stderr) streams and map them to the 'info' and 'error' log levels, respectively, within Loki.

Promtail Configuration for Log Level Relabeling

The following Promtail configuration snippet utilizes pipeline stages to achieve this relabeling. This is crucial for organizing and filtering logs effectively in Loki.

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /var/lib/promtail/positions.yaml
clients:
  - url: https://<LOKI_USERNAME>:<LOKI_PASSWORD>@<LOKI_FQDN>/loki/api/v1/push

scrape_configs:
  - job_name: nginx-info
    static_configs:
    - targets:
        - localhost
      labels:
        job: prod/nginx
        environment: production
        host: demo-app-prod
        level: info
        service_name: demo-app-prod
        __path__: /var/log/nginx/access.log

    pipeline_stages:
    # capture stdout|stderr to level
    - regex:
        expression: '(?P<level>(stdout|stderr))'
    # rename stdout to info, stderr to error
    - template:
        source: level
        template: '{{ if eq .Value "stdout" }}{{ Replace .Value "stdout" "info" -1 }}{{ else if eq .Value "stderr" }}{{ Replace .Value "stderr" "error" -1 }}{{ else if eq .Value "errorstderr" }}{{ Replace .Value "errorstderr" "error" -1 }}{{ .Value }}{{ end }}'
    # set the renamed values to level label
    - labels:
        level:

Understanding the Pipeline Stages

1. Regex Stage: Capturing Log Streams

The first pipeline stage uses a regular expression to identify and capture the log stream type. The expression '(?P<level>(stdout|stderr))' looks for either "stdout" or "stderr" within the log line and assigns it to a temporary field named level.

2. Template Stage: Renaming Log Levels

The second stage, a template stage, takes the captured level value. It uses Go templating to conditionally rename the values:

If the value is "stdout", it's replaced with "info".
If the value is "stderr", it's replaced with "error".
It also includes a condition for "errorstderr" to be mapped to "error".

This ensures that logs originating from stdout are treated as informational, and those from stderr are treated as errors.

3. Labels Stage: Applying Relabeled Values

Finally, the labels stage applies the relabeled value from the level field to the actual log label named level. This makes the relabeled information available for filtering and querying in Loki.

Benefits of Relabeling Log Levels

Improved Log Filtering: Easily filter logs by 'info' or 'error' levels, making it simpler to find relevant information.
Structured Logging: Enforces a consistent log level structure across different applications and services.
Efficient Monitoring: Enables more effective alerting and dashboarding based on critical error logs.