Introduction
Grafana as Code means that all of Grafana’s configuration — including dashboards — is declared by code. The benefits of declaring by code are as follows.
1. all code can be versioned. This means that Code Review is possible, which means that we get all the benefits of Code Review.
2. reduce the amount of manual configuration. If an environment has 30 dashboards and there are 3 environments, you have to manually configure 30*3 dashboards.
3. painless rebuild. It is to rebuild a new set of Grafana, which is stress-free for your team.
4. Easy integration with other systems. For example, according to the business system, automatically generate a new configuration.
5. Can unit test the code of dashboard.
This article is a summary of some of our practical experience.
Overall steps
We look at the configuration of Grafana can be divided into two parts.
- Deployment configuration part: the configuration of the process of deploying Grafana application, such as the application memory size, the domain name used, plug-in installation, etc.. In short, an “empty” Grafana.
- Application configuration part: the configuration of the Grafana application itself, such as the list of folders included, dashboard, LDAP configuration, and so on.
The deployment configuration part, to a large extent, depends on your deployment method. Our recommendation is to run it via docker. If you are not yet live with Kubernetes, then we recommend that you deploy using docker-compose. As for Kubernetes, we recommend that you deploy it using Helm. Chart recommends the official chart: https://github.com/ grafana/helm-charts/tree/main/charts/grafana . Both of them support deployment by configuration.
When deploying with Helm, we have the following recommendations.
1. specify the version of Grafana’s image instead of using the latest version. The consequence of using latest is that some day, your Grafana may not be available because the Grafana version does not match the plugin version.
2. Automate Grafana download dependencies by configuring the plugins field in Values. If your deployment environment has network issues, it may require you to modify the chart to download the plugins from the specified place.
3. Although the configuration of data source and dashboard is provided in Values, we recommend to put it in the application configuration section.
And the application configuration section, the configuration of the Grafana application itself, we recommend to implement it through Terraform, which is mostly considered as an infrastructure tool. In fact, it is a declarative state management tool. In this article, we use it to manage the state of Grafana.
The steps for deploying Grafana configuration using Terraform are as follows.
1. Prepare the backend for storing Terraform state data. We use PostgreSQL as the backend.
2. in the editor, write the Terraform configuration code.
3. write the Grafana dashboard via Jsonnet.
4. building out the main.tf.json file via Jsonnet.
5. deploy Grafana configuration via terraform.
Of course, the prerequisite for performing the above steps is that
1. Terraform, Jsonnet must be installed.
2. the PostgreSQL service is ready.
3. VSCode or other code editor of your choice.
4. a pre-deployed Grafana service.
Prepare the backend to store Terraform state data
Terraform supports a variety of backends to store state data, including: artifactory, azurerm, consul, cos, etcd, gcs, oss, s3, pg. This article uses pg, which is PostgreSQL.
In PostgreSQL creation in DB.
createdb terraform_backendterraform_backend is the name of the DB. You can give different names according to different business scenarios. And we can also create different DB for different business scenarios. for example, the DB for SIT environment is called terraform_backend_sit, and the DB for production environment is called terraform_backend_prod.
Writing Terraform Code
Terraform uses a configuration language called HCL by default. Also, it supports configuration using JSON. Here, we recommend using JSON as Terraform’s configuration language. This is because Jsonnet can easily generate JSON, but cannot easily generate HCL configurations.
The following is a framework for writing Terraform configuration using Jsonnet, similar to the role of HelloWord. This configuration file is the entry file for Terraform , the file name used main.tf.jsonnet .
You can add your code to this framework to.
local grafanaDashboardJsonContent = import 'grafanaDashboardJsonContent.libsonnet';
{
terraform: {
required_providers: {
// Specify the version of Grafana's Provider
grafana: {
source: 'grafana/grafana',
version: '1.21.1',
},
},
backend: {
pg: {
conn_str: 'postgres://postgres:postgres@127.0.0.1:5432/grafana_terraform_backend?sslmode=disable',
},
},
},
provider: {
grafana: {
// The address of the grafana you wish to configure
url: 'grafanas url',
// https://registry.terraform.io/providers/grafana/grafana/latest/docs#auth
auth: '%(name)s:%(password)s' % { name: admin, password: adminpassword }
},
},
resource: {
grafana_folder: {
abc_team: {
title: 'abc Team'
}
},
// Define the data source
grafana_data_source: {
influxdb1:{
type: "influxdb",
name: "myapp-metrics",
url: "http://influxdb.example.net:8086/",
username: "myapp",
password: "foobarbaz",
database_name = "dbname"
}
// other datasources can be defined below
},
// Jsonnet syntax to create teams in bulk based on the definitions in the teams variable. teams can be provided by other Jsonnet files
grafana_team: {
[t]: { name: teams[t].name }
for t in std.objectFields(teams)
},
grafana_dashboard: {
// jvm_dashboard is the dashboard's identifier in Jsonnet. An object is accepted.
// grafanaDashboardJsonContent is defined in other Jsonnet files and is imported into this file.
jvm_dashboard: grafanaDashboardJsonContent
}
},
}
}Writing Dashboard via Jsonnet
Usually we have one Dashboard with one Jsonnet file. Then we introduce it into main.tf.jsonnet entry file by import. The way to import is as follows.
local grafanaDashboardJsonContent = import 'grafanaDashboardJsonContent.libsonnet';grafanaDashboardJsonContent.libsonnet reads, in part, as follows:
local grafana = import 'grafonnet/grafana.libsonnet';grafana.dashboard.new(
'JVM',
refresh='1m',
time_from='now-1h',
tags=['java']
).addRow(
row.new(
title='Memory',
height='250px',
)
.addPanel(
graphPanel.new(
'JVM Memory Heap',
span=6,
format='bytes',
fill=0,
min=0,
decimals=2,
datasource='-- Mixed --',
legend_values=true,
legend_min=true,
legend_max=true,
legend_current=true,
legend_total=false,
legend_avg=true,
legend_alignAsTable=true,
legend_sideWidth=200,
)
)
)
The finished content can be found in the official sample: https://github.com/grafana/grafonnet-lib/blob/master/examples/jvm.jsonnet
Jsonnet has very good modularity. grafana exposes its own Jsonnet library for writing Grafana’s Dashboard https://github.com/grafana/grafonnet-lib . So, in the above code, on the first line, we introduce it.
local grafana = import 'grafonnet/grafana.libsonnet';Note: import picks up the relative path.
So far for this, the directory structure of the whole project is as follows.
.
├── grafanaDashboardJsonContent.json
├── grafonnet
│ └── grafana.libsonnet
└── main.tf.jsonYou can also create a dashboards folder to store all the dashboards. to make it easier to manage the dashboards.
Build out main.tf.json file via Jsonnet
Execute the command
jsonnet main.tf.jsonnet > main.tf.jsonNote that the generated files must start with . The terraform command will load all files in the directory with the .tf.json suffix.
Deploying Grafana configuration via Terraform
Terraform provides a standard process to implement deployment. The steps are as follows (all executed at the root of the project).
1. terraofm init : initialize the project, including downloading the provider, backend storage.
2. terraform plan: check the differences with the remote configuration (status) (optional). For example, this example is to see the difference between your local configuration and the running Grafana service.
3. terraform apply: perform the deployment.
With the above steps, we have deployed the Grafana configuration to the Grafana service.
Automate all
The above is just a description of the process of configuring Grafana by code. However, it is semi-automated. We have to manually execute at least jsonnet command and terraform command. As for, how to design and implement the automation flow is not the focus of this article, so we will not expand on this article.
How to avoid writing Grafana Dashboard from scratch
Not all Grafana is built from scratch. Many companies had Grafana before Grafana as Code existed. At this point, if we were to write the Grafana Dashboard all from scratch, the workload would be very high.
The solution is to pull down all the Dashboard’s Json files directly from the existing Grafana service, and then change the suffix to Jsonnet, which can be referenced by our main.tf.jsonnet file.
This process, though, is also manual, but it has been much easier than writing it from scratch.
The specific steps are as follows.
1. download the Dashboard json file through Grafana’s API. API documentation: https:/ /grafana.com/docs/grafana/latest/http_api/dashboard/#get-dashboard-by-uid.
2. save the json as a jsonnet file in the directory.
3. main.tf.jsonnet introduces this jsonnet.
There is a pitfall here that you need to pay attention to, that is, the json file of Dashboard downloaded through API and the JSON model of Dashboard are two different things. You can’t use the JSON model as a parameter passed to the main.tf.jsonnet. Terraform’s grafana provider uses the Grafana API to configure the Grafana service.
The following figure shows a Dashboard JSON Model, which cannot be used as a parameter.
Summary
The “Code” in Grafana as Code is divided into two parts: the deployment code and the configuration code. Deployment code is related to the deployment method and is not covered in this article.
Configuration code refers to the configuration of Grafana services through code. The principle is to manage the state of Grafana services through Terraform. And HCL, as the configuration language of Terraform, is not consistent with our Jsonnet language. So, we use Jsonnet to generate the configuration in json format that Terraform also recognizes. Of course, you can also generate configurations in the formats supported by Terraform through other configuration languages.
