This document provides a configuration example for Promtail, focusing on how to drop specific loglines to optimize data ingestion into Loki. By strategically filtering logs, you can reduce noise, save storage, and improve query performance.

Promtail Configuration for Dropping Loglines

The following YAML configuration demonstrates various methods to drop log entries based on different criteria. This includes dropping logs based on their content, age, and length, as well as specific patterns like health checks or noisy error messages.

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /var/lib/promtail/positions.yaml
clients:
  - url: https://<LOKI_USERNAME>:<LOKI_PASSWORD>@<LOKI_FQDN>/loki/api/v1/push

scrape_configs:
  - job_name: nginx-info
    static_configs:
    - targets:
        - localhost
      labels:
        job: prod/nginx
        environment: production
        host: demo-app-prod
        level: info
        service_name: demo-app-prod
        __path__: /var/log/nginx/access.log

    pipeline_stages:
    # Drop loglines matching specific URL patterns (e.g., health checks)
    # https://grafana.com/docs/loki/latest/clients/promtail/stages/drop/
    - drop:
        expression: "(.*/health-check.*)|(.*/health.*)"
    # Drop loglines older than 24 hours to reduce historical data ingestion
    - drop:
        older_than: 24h
        drop_counter_reason: "line_too_old"
    # Drop loglines that exceed a certain length to prevent oversized entries
    - drop:
        longer_than: 8kb
        drop_counter_reason: "line_too_long"
    # Drop logs containing specific "noisy" keywords using a match stage
    # https://grafana.com/docs/loki/latest/clients/promtail/stages/match/
    - match:
        selector: '{app="promtail"} |~ ".*noisy error.*"'
        action: drop
        drop_counter_reason: promtail_noisy_error
    # Drop logs from specific Loki components or applications
    - match:
        selector: '{app="loki", component="gateway"}'
        action: drop
        drop_counter_reason: loki_gateway_logs
    # Drop informational logs from Loki queriers to reduce verbosity
    - match:
        selector: '{app="loki", component="querier"} |= "level=info"'
        action: drop
        drop_counter_reason: loki_querier_info_logs
    # Example: Drop ELB health checker logs specifically for the nginx job
    # https://github.com/cyriltovena/loki/blob/master/docs/clients/promtail/stages/match.md#example
    - match:
        pipeline_name: 'drop_elb_healthchecks'
        selector: '{job="prod/nginx"} |= "ELB-HealthChecker"'
        action: drop

Understanding Promtail Drop Stages

Promtail's pipeline stages offer powerful ways to process logs before they are sent to Loki. The drop stage allows you to remove log lines based on regular expressions, age, or length. The match stage, when used with the action: drop, provides more granular control by allowing you to drop logs based on label selectors and content matching.

Key Drop Configurations Explained:

• Dropping by Expression: Use regular expressions to filter out logs with specific patterns, such as health check endpoints.
• Dropping by Age: The older_than parameter helps discard logs that are no longer relevant, saving storage space.
• Dropping by Length: The longer_than parameter prevents excessively large log entries from being ingested.
• Dropping by Content Match: The match stage with a content selector (e.g., |~ ".*noisy error.*") is effective for removing specific, unwanted messages.
• Dropping by Labels: Filter logs based on their associated labels, useful for excluding logs from certain components or environments.

Optimizing Log Ingestion with Promtail

Implementing effective log dropping strategies is crucial for maintaining an efficient and cost-effective logging infrastructure with Loki. By carefully configuring Promtail, you can ensure that only valuable log data is stored and analyzed, leading to faster insights and reduced operational overhead.

For more advanced filtering and processing, explore other Promtail pipeline stages such as regex, template, and labels.

Further Resources:

• Promtail Official Documentation
• Loki Configuration Guide
• Promtail Pipeline Stages