Loki: A Log Aggregation System

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It indexes metadata about logs rather than the full log content, making it cost-effective and simple to operate.

Key Resources for Loki

Explore these resources to deepen your understanding and implementation of Loki:

Grafana Dashboards for Loki

Enhance your monitoring experience with pre-built dashboards for Loki:

Loki Overview Dashboard

Getting Started with Loki

Set up your Loki stack efficiently with these guides:

For a distributed Loki stack setup, consider this resource:

Distributed Loki Stack with MinIO

Configuring Logging Clients for Loki

Integrate your applications and services with Loki using various logging clients. The Loki Docker driver is a popular choice for containerized environments.

Installing the Loki Docker Driver

Install the Loki Docker driver to enable your Docker containers to send logs directly to Loki:

$ sudo docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Using the Loki Driver in Docker Compose

Configure your docker-compose.yml to direct logs from specific services to your Loki instance:

version: '3.7'
services:
  website:
    image: nginx
    container_name: website
    restart: unless-stopped
    logging:
      driver: loki
      options:
        loki-url: http://192.168.0.4:3100/loki/api/v1/push
        loki-external-labels: job=dockerlogs,stack=nginx
        loki-pipeline-stages: |
          - regex:
              expression: '(level|lvl|severity)=(?P<level>\w+)'
          - labels:
              level:

Setting Loki as the Default Log Driver

To have all containers log to Loki by default, configure the Docker daemon's daemon.json file:

$ cat /etc/docker/daemon.json
{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://docker:x@loki.x.x.x.x/loki/api/v1/push",
        "loki-batch-size": "300",
        "loki-external-labels": "job=dev/dockerlogs,container_name={{.Name}},cluster_name=dev-ecs-cluster,hostname=ip-172-31-50-37.eu-west-1.compute.internal,aws_account=dev,environment=development"
    }
}

After modifying daemon.json, restart the Docker service:

$ sudo systemctl restart docker

You can test this default configuration by running a container without explicitly setting the log driver:

$ docker run --rm -it --name foobar12 alpine echo hi

The logs from this container will be sent to Loki.

If the daemon.json is not configured for default logging, you can specify the Docker logging driver per container:

$ docker run --rm -it --log-driver loki --log-opt loki-url="https://x:x@loki.domain.com/loki/api/v1/push" --log-opt loki-external-labels="job=debug/dockerlogs" hello-world

Observe your logs within Loki:

Loki Logs Example

Using LogCLI for Log Exploration

LogCLI is a powerful command-line interface for querying and exploring logs stored in Loki. It allows you to efficiently search, filter, and analyze your log data directly from your terminal.

Refer to the LogCLI Cheatsheet to learn how to effectively use this tool for your log management needs.