This document outlines the configuration for Promtail to leverage AWS EC2 Service Discovery for dynamic log collection. By integrating Promtail with EC2's metadata, you can automatically discover and scrape logs from your EC2 instances without manual configuration updates.

Promtail EC2 Service Discovery Configuration

The following YAML configuration demonstrates how to set up Promtail for EC2 service discovery. This setup allows Promtail to dynamically find EC2 instances based on their tags and metadata, and then scrape logs from specified paths.

# https://grafana.com/docs/loki/latest/clients/promtail/scraping/
# https://grafana.com/blog/2020/07/13/loki-tutorial-how-to-set-up-promtail-on-aws-ec2-to-find-and-analyze-your-logs/
server:
  http_listen_port: 3100
  grpc_listen_port: 0

clients:
  - url: https://user:pass@loki.domain.com/loki/api/v1/push

positions:
  filename: /opt/promtail/positions.yaml

scrape_configs:
  - job_name: prod/ec2-logs
    ec2_sd_configs:
      - region: eu-west-1
        #access_key: REDACTED
        #secret_key: REDACTED
        #role_arn: arn:aws:iam::000000000000:role/PrometheusEC2DynamicScrapeRole
    relabel_configs:
      - source_labels: [__meta_ec2_architecture]
        regex: "(.*)"
        replacement: "prod/server-logs"
        target_label: job
      - source_labels: [__meta_ec2_tag_Name]
        target_label: name
        action: replace
      - source_labels: [__meta_ec2_instance_id]
        target_label: instance
        action: replace
      - source_labels: [__meta_ec2_availability_zone]
        target_label: zone
        action: replace
      - action: replace
        replacement: /var/log/**.log
        target_label: __path__
      - source_labels: [__meta_ec2_private_dns_name]
        regex: "(.*)\\.(.*)\\.compute\\.internal"
        replacement: '${1}'
        target_label: __host__

  - job_name: prod/journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      labels:
        job: prod/systemd-journal
        name: my-ec2-instance
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal__hostname']
        target_label: __host__
      - source_labels: ['__journal_syslog_identifier']
        target_label: syslog_identifier

# prod/ec2-logs produces:
#  job="prod/server-logs"
#  instance="i-00000000000"
#  name="my-ec2-instance"
#  zone="eu-west-1a"
#  filename="/var/log/auth.log"

# prod/journal produces:
#  job="prod/systemd-journal"
#  name="my-ec2-instance"
#  syslog_identifier="promtail"
#  unit="promtail.service"

Understanding EC2 Service Discovery

The ec2_sd_configs section enables Promtail to query AWS EC2 for instance information. It uses the specified AWS region to discover instances. For security, it's recommended to use IAM roles (role_arn) rather than hardcoding access keys.

Relabeling for Log Identification

The relabel_configs are crucial for transforming the discovered EC2 metadata into labels that Promtail uses to identify and route logs. This includes setting the job name based on instance architecture, adding name and instance labels from EC2 tags and IDs, and defining the log file path (__path__).

Journald Log Collection

In addition to EC2 instance logs, the configuration includes a separate job for collecting logs from systemd-journald. This ensures that system-level logs are also captured and sent to Loki.

External Resources

• Promtail Official Documentation
• AWS EC2 Tagging Documentation
• Loki Relabeling Documentation