Promtail Configuration for Java and Syslog

This example demonstrates a Promtail configuration designed to collect both system logs (syslog) and application-specific Java logs from a Linux operating system. It's structured for a production environment, using clear job names, environments, and host labels for easy identification and filtering within Loki.

Key Configuration Components

The configuration is divided into several key sections:

• Server Settings: Defines the ports Promtail listens on.
• Positions: Specifies where Promtail stores its state to resume collection after restarts.
• Clients: Configures the connection details to your Loki instance, including authentication.
• Scrape Configurations: This is the core part, defining how Promtail discovers and scrapes logs.

Syslog Collection

The syslog job targets the standard system log file (/var/log/syslog). It assigns labels like job: prod/syslog, host, and environment to categorize these logs effectively.

Java Application Log Collection

The myapp job is configured to collect logs for a Java application. It uses a wildcard pattern (/var/log/myapp/myapp-logs_*.log) to capture multiple log files. Crucially, it sets a service_name label, which is vital for distinguishing logs from different applications.

Pipeline Stages for Log Processing

The myapp job also includes pipeline_stages to process logs before they are sent to Loki. This example shows how to:

• Match: Select logs based on specific labels (e.g., service_name="myapp-prod" and environment="production").
• Regex: Extract the log level (INFO, WARNING, ERROR) from log lines using a regular expression.
• Template: Convert the extracted log level to lowercase (e.g., INFO to info).
• Labels: Assign the processed log level as a new label (level) to the log entry. This allows for powerful filtering and querying in Loki based on log severity.

This structured approach to log collection and processing ensures that your Java application logs are not only stored but also enriched with metadata, making them easier to analyze and troubleshoot.

External Resources

• Promtail Documentation
• Loki Configuration Guide
• Syslog Protocol Specification (RFC 5424)

# Example: promtail to collect syslog and java logs from linux os
# Application called myapp running in production
#
# job: prod/myapp
# environment: production
# host: myapp-prod.domain (or hostname)
# service_name: myapp-prod

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /var/lib/promtail/positions.yaml
clients:
  - url: https://<LOKI_USERNAME>:<LOKI_PASSWORD>@<LOKI_FQDN>/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    pipeline_stages:
    static_configs:
    - targets:
        - localhost
      labels:
        job: prod/syslog
        host: myapp-prod.domain
        environment: production
        __path__: /var/log/syslog

  - job_name: myapp
    static_configs:
    - targets:
        - localhost
      labels:
        job: prod/myapp
        environment: production
        host: myapp-prod.domain
        service_name: myapp-prod
        __path__: /var/log/myapp/myapp-logs_*.log

    # remaps INFO to info for specified selector
    pipeline_stages:
    # https://github.com/cyriltovena/loki/blob/master/docs/clients/promtail/stages/match.md#example
    - match:
        selector: '{service_name="myapp-prod",environment="production"}'
        # selector: '{service_name="myapp-prod",environment="production"} |~ "GET|POST"' <- if you only want specific logs to be matched by the pipeline stage
        stages:
        - regex:
            expression: "(?P<level>(INFO|WARNING|ERROR))(.*)"
        - template:
            source: level
            template: '{{ ToLower .Value }}'
        - labels:
            level: