datadog_logs
Configure Datadog logs for your ECS Fargate services with this comprehensive JSON example. Learn how to set up log drivers, environment variables, and secrets for effective log collection.
Datadog Logs Configuration for ECS
This document provides a sample AWS ECS task definition for configuring Datadog logs. It demonstrates how to integrate Datadog's logging capabilities with your containerized applications running on AWS Elastic Container Service (ECS), particularly with Fargate.
Understanding Datadog Log Configuration in ECS
Effective log management is crucial for monitoring and debugging applications. This configuration leverages AWS FireLens, a container log router for Amazon ECS and AWS Fargate, to send logs to Datadog. FireLens uses Fluent Bit or Fluentd as the log router, allowing for flexible log processing and routing.
Key Components of the Configuration
The provided JSON defines an ECS task. Here are the essential parts related to Datadog logging:
Container Definitions
The task definition includes multiple container definitions:
- Application Container (e.g.,
eng-my-service-api
): This is your primary application. It's configured to send logs to the log router. - Datadog Agent Container (
datadog
): This container runs the Datadog Agent, responsible for collecting and forwarding logs and metrics to Datadog. - Log Router Container (
log_router
): This container, typically usingaws-for-fluent-bit
, acts as the intermediary, collecting logs from other containers and routing them to the specified destination (Datadog in this case).
Log Drivers and Options
The configuration specifies different log drivers:
awsfirelens
: Used for the application container to route logs through FireLens.awslogs
: Used for the Datadog Agent container to send its own logs to AWS CloudWatch Logs for debugging.
Specific options within the logConfiguration
block define how logs are handled, including the Datadog endpoint, service name, source, and tags.
Environment Variables and Secrets
Environment variables are used to configure the Datadog Agent and application, such as enabling Datadog features (DD_ENABLED
, DD_LOGS_ENABLED
), setting the Datadog site (DD_SITE
), and specifying tags (DD_TAGS
). Secrets, such as the Datadog API and App keys, are securely managed using AWS Systems Manager Parameter Store.
Example Task Definition JSON
{
"family": "my-service",
"executionRoleArn": "arn:aws:iam::xxxxxxxxxxxx:role/ecs-my-service-execution-role",
"taskRoleArn": "arn:aws:iam::xxxxxxxxxxxx:role/ecs-my-service-task-role",
"cpu": "512",
"memory": "1024",
"containerDefinitions": [
{
"name": "eng-my-service-api",
"image": "xxxxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/my-service-api:latest",
"essential": true,
"cpu": 0,
"memoryReservation": 128,
"portMappings": [
{
"hostPort": 8080,
"protocol": "tcp",
"containerPort": 8080
}
],
"environment": [
{
"name": "AWS_REGION",
"value": "eu-west-1"
},
{
"name": "DD_ENABLED",
"value": "true"
},
{
"name": "DD_HOST",
"value": "https://api.datadoghq.eu"
},
{
"name": "DD_TAGS",
"value": "env:dev"
}
],
"secrets": [
{
"valueFrom": "arn:aws:ssm:eu-west-1:xxxxxxxxxxxx:parameter/my-service-api/dev/DD_API_KEY",
"name": "DD_API_KEY"
},
{
"valueFrom": "arn:aws:ssm:eu-west-1:xxxxxxxxxxxx:parameter/my-service-api/dev/DD_APP_KEY",
"name": "DD_APP_KEY"
}
],
"logConfiguration": {
"logDriver": "awsfirelens",
"secretOptions": [
{
"valueFrom": "arn:aws:ssm:eu-west-1:xxxxxxxxxxxx:parameter/my-service-api/dev/DD_API_KEY",
"name": "apikey"
}
],
"options": {
"Name": "datadog",
"Host": "http-intake.logs.datadoghq.eu",
"provider": "ecs",
"dd_message_key": "log",
"dd_service": "my-service-api",
"dd_source": "fargate",
"dd_tags": "project:my-service-api"
}
}
},
{
"name": "datadog",
"image": "datadog/agent:7",
"essential": true,
"cpu": 10,
"environment": [
{
"name": "DD_APM_ENABLED",
"value": "true"
},
{
"name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
"value": "true"
},
{
"name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
"value": "true"
},
{
"name": "DD_LOGS_ENABLED",
"value": "true"
},
{
"name": "DD_SITE",
"value": "datadoghq.eu"
},
{
"name": "ECS_FARGATE",
"value": "true"
}
],
"secrets": [
{
"valueFrom": "arn:aws:ssm:eu-west-1:xxxxxxxxxxxx:parameter/my-service-api/dev/DD_API_KEY",
"name": "DD_API_KEY"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/datadog",
"awslogs-create-group": "true",
"awslogs-region": "eu-west-1",
"awslogs-stream-prefix": "my-service-api"
}
}
},
{
"name": "log_router",
"essential": true,
"cpu": 0,
"image": "906394416424.dkr.ecr.eu-west-1.amazonaws.com/aws-for-fluent-bit:latest",
"firelensConfiguration": {
"type": "fluentbit"
}
}
],
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc"
}