This document outlines the configuration for collecting container metrics using cAdvisor with Prometheus, specifically focusing on AWS EC2 integration.

Prometheus Configuration for cAdvisor

To effectively monitor your containers, Prometheus needs to be configured to scrape metrics exposed by cAdvisor. This section details the necessary scrape_configs, particularly when leveraging AWS EC2's service discovery capabilities.

Key Labels for Container Metrics

When collecting container metrics, it's crucial to tag your instances appropriately. The following labels are commonly used and can be configured in Prometheus:

• instance: Typically maps to the EC2 instance name.
• environment: Identifies the deployment environment (e.g., production, staging).
• cluster_name: Specifies the Kubernetes or ECS cluster the container belongs to.

AWS EC2 Service Discovery Configuration

For AWS environments, Prometheus can dynamically discover targets using EC2 service discovery. This is particularly useful for managing ephemeral instances and ensuring Prometheus always scrapes the latest running cAdvisor instances.

Below is an example of a Prometheus scrape_configs block configured for AWS:

scrape_configs:
  - job_name: container-metrics
    scrape_interval: 15s
    ec2_sd_configs:
    - region: eu-west-1
      role_arn: 'arn:aws:iam::000000000000:role/prometheus-ec2-role'
      port: 9100
      filters:
        - name: tag:PrometheusContainerScrape
          values:
            - Enabled
    relabel_configs:
    - source_labels: [__meta_ec2_private_ip]
      replacement: '${1}:8080'
      target_label: __address__
    - source_labels: [__meta_ec2_tag_Name]
      target_label: instance
    - source_labels: [__meta_ec2_tag_ECSClusterName]
      target_label: cluster_name
    - source_labels: [__meta_ec2_tag_Environment]
      target_label: environment

Explanation of Configuration Parameters

• job_name: container-metrics: Assigns a name to this scraping job.
• scrape_interval: 15s: Sets how frequently Prometheus should scrape metrics.
• ec2_sd_configs: Configures EC2 service discovery.
  • region: The AWS region to scan for instances.
  • role_arn: The IAM role ARN that Prometheus assumes to access EC2 information.
  • port: The port cAdvisor is listening on (commonly 9100 for node_exporter, but cAdvisor might be on a different port, adjust as needed).
  • filters: Allows filtering EC2 instances based on tags. Here, it selects instances tagged with PrometheusContainerScrape=Enabled.
• relabel_configs: Rules to modify labels before ingestion.
  • The first rule sets the __address__ label to the private IP and the cAdvisor port (8080 in this example).
  • Subsequent rules map EC2 tags (Name, ECSClusterName, Environment) to Prometheus labels (instance, cluster_name, environment).

Further Resources

• Prometheus Scrape Config Documentation
• Prometheus EC2 Service Discovery
• cAdvisor GitHub Repository