Grafana Cheatsheet - Monitoring & Visualization Guide

This cheatsheet provides quick reference for using Grafana for monitoring and data visualization. It covers essential aspects from dashboards and tutorials to specific datasource configurations and query examples.

Grafana Overview and Navigation

• Dashboards
• Tutorials

Datasource Configurations

CloudWatch Datasource

• CloudWatch Datasource Setup
• CloudWatch Variables
• CloudWatch Queries

Elasticsearch Datasource

• Elasticsearch Datasource Setup
• Elasticsearch Variables

MySQL Datasource

• MySQL Datasource Setup
• MySQL Variables
• MySQL Queries

Prometheus Datasource

• Prometheus Datasource Setup
• Prometheus Variables
• Prometheus Queries
• Prometheus Tables

Loki Datasource

• Loki Datasource Setup
• Loki Variables
• Loki Queries

Dashboards

Discover pre-built dashboards to accelerate your monitoring setup.

• Loki Dashboard Example
• JVM Micrometer Dashboard

Tutorials and Resources

Explore helpful tutorials and documentation to deepen your Grafana knowledge.

• sbcode Grafana Tutorials
• Official Grafana Documentation

Datasource: CloudWatch

Configure and query AWS CloudWatch metrics within Grafana.

Variables for CloudWatch

Utilize CloudWatch variables to dynamically filter and select data in your dashboards.

Documentation:

• Grafana CloudWatch Datasource Docs
• Grafana CloudWatch Query Variables

Overview:

The Name field allows you to use it as a variable, example:

Name: region
Type: Query
Label: Region
Query: regions()

This will display "Region" in Grafana, and you can use a variable $region to select the value from the Region selector.

AWS Regions:

Query: regions()

AutoScaling Group:

Query: dimension_values($region,AWS/EC2,CPUUtilization,AutoScalingGroupName)

EC2 Instance Tag Names:

Name: instancename
Query: ec2_instance_attribute($region, Tags.Name, {})

EC2 Instance ID from Tag Name:

Name: instanceid
Query: ec2_instance_attribute($region, InstanceId, {"tag:Name": ["$instancename"]})

EC2 InstanceId from Tag Name (filtered):

Query: ec2_instance_attribute(eu-west-1, Tags.Name, {"tag:Name":["prod-*"]}) 

EC2 InstanceId from Tag Name (filtered):

Query: ec2_instance_attribute(eu-west-1, InstanceId, {"tag:ASG":["my-app-asg"]}) 

EBS VolumeId from InstanceId:

Query: ebs_volume_ids($region, $instanceid)

ECS Cluster Name:

Query: dimension_values($region,AWS/ECS,CPUUtilization,ClusterName)

ECS Service Name:

Query: dimension_values($region,AWS/ECS,CPUUtilization,ServiceName)

RDS Cluster Name:

Query: dimension_values($region,AWS/RDS,CPUUtilization,DBClusterIdentifier)

RDS Instance Name:

Query: dimension_values($region,AWS/RDS,CPUUtilization,DBInstanceIdentifier)

CloudWatch Logs, LogGroup Names:

Query: dimension_values($region,AWS/Logs, IncomingBytes,LogGroupName)

CloudWatch Queries

Leverage CloudWatch Logs Query Syntax for advanced log analysis.

Using CloudWatch Logs Query Syntax:

fields @timestamp, @message, @log, @logStream
| sort @timestamp desc
| limit 30

Datasource: Elasticsearch

Integrate Elasticsearch for log analysis and time-series data visualization.

• Effective Elasticsearch Usage in Grafana

Variables for Elasticsearch

Create dynamic variables for Elasticsearch queries.

Domain Name:

{"find": "terms", "field": "domain_name.keyword"}

Domain Name (Filtered Results):

{"find": "terms", "field": "domain_name.keyword", "query": "domain_name: *.mydomain.com"}

2 Variables, ALB and Domain Name (domain name results filtered based on the alb that you select / should be in correct order):

alb_name:
{"find": "terms", "field": "alb_name.keyword"}

domain_name:
{"find": "terms", "field": "domain_name.keyword", "query": "domain_name: *.mydomain.com AND alb_name:$alb_name.keyword"}

Datasource: MySQL

Connect Grafana to MySQL databases for data visualization.

Variables for MySQL

Define variables for dynamic MySQL queries.

Name: status Label: Status Type: Query Datasource: MySQL Query: SELECT status FROM mytable

To use a regex to filter out any NULLs:

# this will only return results with letters/numbers
/([a-zA-Z0-9\.]+)/  

Queries for MySQL

Examples of Grafana queries for MySQL panels.

Gauge:

SELECT 
country,
SUM(cnt) AS total,
NOW() AS time
FROM mytable
WHERE status = ${status}
GROUP BY country

Bar Gauge:

SELECT NOW() AS time, count(*) as cnt, CONCAT(name,', ',surname,', ',country) AS entity FROM mytable 
WHERE status = "PENDING"
AND name REGEXP '${name:pipe}' 
AND surname REGEXP '${surname:pipe}'
AND country REGEXP '${country:pipe}'
GROUP BY CONCAT(name,', ',surname,', ',country)

Table Panel:

SELECT name, surname, country, status, pending_time from mytable
WHERE status = "PENDING"
AND name REGEXP '${name:pipe}' 
AND surname REGEXP '${surname:pipe}'
AND country REGEXP '${country:pipe}'

Prometheus: Datasource

Integrate Prometheus for time-series monitoring data.

Variables for Prometheus

Create dynamic variables for Prometheus queries.

Basics

Example: Define a jobs variable based on the up metric.

up{cluster_name="cluster-a",instance="1.1.1.1:443",job="container-metrics"}
up{cluster_name="cluster-b=",instance="1.1.1.1:443",job="node-metrics"}

Add a variable with the following configuration:

Name: jobs
Label: Jobs
Query: label_values(up, job)
Datasource: Prometheus

This will produce container-metrics and node-metrics. In your dashboard query, you can select them using:

up{job=~"$job"}

Filtered Variables

Example: Display jobs only from cluster-b and name the variable cluster_b_jobs.

label: cluster_b_jobs
label_values(up{cluster_name="cluster-b"}, job)

You can use this variable to filter further, e.g., label_values(metric{jobs=~"$cluster_b_jobs"}, some_label). This is useful for getting environments and then filtering on resources.

Regex Filtering

Example: Filter job results to display only those matching a pattern.

Current job results:

qa/nginx
qa/app
qa/app-syslog
qa/app-deploy
prod/app
prod/app-syslog
prod/app-deploy

To display {env}/app, use the following query and regex:

Query:

label_values(labels, job)

Regex:

/^(.*app)/

This results in:

qa/app
prod/app

To get everything after the /:

/.*(.*app.*).*/

Which will result in:

app
app-syslog 
app-deploy

For an example where you want to return everything up until the numbers:

ecs-prod-app-10-container-12345
ecs-dev-app-12-container-12345

You can use:

/^(.*?)-[0-9]/

Which will result in:

ecs-prod-app
ecs-dev-app

For Kubernetes namespaces:

Name: container Label: container Query: kube_pod_container_info{namespace="$namespace"} Regex: /.*container="([^"]*).*/ Datasource: Prometheus

For Kubernetes containers:

Name: namespace Label: namespace Query: query_result(kube_namespace_labels) Regex: /.*namespace="([^"]*).*/ Datasource: Prometheus

Queries for Prometheus

Explore common Prometheus query patterns for Grafana panels.

Tables for Prometheus

Utilize data links in Grafana to create interactive tables and link to external resources.

Using datalinks in Grafana, to parse values in your url you can use $__cell_0 and if you need to do some formatting you can use ${__cell_3:raw}, like below:

https://gitlab.com/${__cell_3:raw}/-/pipelines/$__cell_7

Resources:

• Grafana Data Links Documentation
• Grafana Advanced Variable Formatting

Loki Datasource

Integrate Loki for log aggregation and querying.

Variables for Loki

Create dynamic variables for Loki queries.

Basics: Jobs

Example: Define a jobs variable for Loki logs.

label_values({job=~".+"}, job)

Add a variable with the following configuration:

Name: job
Label: Jobs
Query: label_values({job=~".+"}, job)
Datasource: Loki

This will produce values like systemd-logs and server-logs. In your dashboard query, you can select them using:

{job=~"$job"}

Basics: Name

Example: Define a name variable for Loki logs.

label_values({job="server-logs"}, name)

Add a variable with the following configuration:

Name: name
Label: Name
Query: label_values({job="server-logs"}, name)
Datasource: Loki

This will produce values like web-server-01 and web-server-02. In your dashboard query, you can select them using:

{name=~"$name"}

Loki Queries

Explore common Loki query patterns for log analysis.