Monitoring with Prometheus

Vagrant

Let’s start with the Vagrantfile. This file is the input for Vagrant and describes the basic structure of our environment. Typically, I have three parts in a Vagrantfile:

1. an array which describes the servers and their attributes
2. some general hypervisor settings
3. the logic to create the servers from the array

Lets take a look at the first part:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|

  # =========================================================================
  # The servers in our environment
  # =========================================================================
  servers=[
    {
      :hostname => "monitoring",
      :box => "geerlingguy/ubuntu1604",
      :ip => "192.168.100.10",
      :port => 50001,
      :ram => 1024,
      :cpu => 1
    },
    {
      :hostname => "app-server-1",
      :box => "geerlingguy/ubuntu1604",
      :ip => "192.168.100.11",
      :port => 50002,
      :ram => 1024,
      :cpu => 1
    },
    {
      :hostname => "app-server-2",
      :box => "geerlingguy/ubuntu1604",
      :ip => "192.168.100.12",
      :port => 50003,
      :ram => 1024,
      :cpu => 1
    }
  ]

You’ll notice that the three servers we mentioned in the previous section get described here. I like the Vagrant Box from geerlingguy, as it works better as the official one from Canonical. I use predefined private IPs (and not DNS), as I feed these IPs later to Ansible. Sometimes I had issues with host port clashes when working with multiple Vagrant environments at the same time, so I usually predefine them as well. From time to time, I need different amount of resources for a multi VM environment, so I added attributes for CPUs and memory to my template as well.

The second part is this:

36
37
38
39
40
41
42

  # =========================================================================
  # The general settings
  # =========================================================================
  config.vm.provider "virtualbox" do |v|
    # https://www.vagrantup.com/docs/virtualbox/configuration.html#linked-clones
    v.linked_clone = true
  end

I use a hypervisor specific feature from VirtualBox to speed things up a little. When you destroy and create such an environment multiple times, this comes in handy.

And the third part is this:

44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

  # =========================================================================
  # Server specific settings
  # =========================================================================
  servers.each do |machine|
    config.vm.define machine[:hostname] do |node|
      node.vm.box = machine[:box]
      node.vm.hostname = machine[:hostname]
      node.vm.network "private_network", ip: machine[:ip]
      node.vm.network "forwarded_port", guest: 22, host: machine[:port], id: "ssh"

      node.vm.provider "virtualbox" do |vb|
        vb.memory = machine[:ram]
        vb.cpus = machine[:cpu]
      end
    end
  end
end

This iterates through the servers defined in the first part and applies all the attributes we defined.

With this file locally in place, you can influence the life cycle of your servers with vagrant up and vagrant destroy. To continue with this post, start the servers with:

1

$ vagrant up          # create the servers

Now we are ready to deploy something onto those servers.

Ansible hosts

I’m going to use more and more Ansible examples in this blog, as this is my tool of choice for such tasks. Doing such things with the shell looks smaller and easier at first, but the more nodes you have, the more Ansible comes in handy. If you haven’t tried it yet, give it a chance, it’s awesome.

Ansible needs a file which specifies its targets to operate on. It uses the ini syntax and you can name it whatever you want, I called it hosts.ini throughout this post. The content is:

1
2
3
4
5
6

[infra-servers]
monitoring ansible_host=192.168.100.10 ansible_user=vagrant ansible_ssh_pass=vagrant

[application-servers]
app-server-1 ansible_host=192.168.100.11 ansible_user=vagrant ansible_ssh_pass=vagrant
app-server-2 ansible_host=192.168.100.12 ansible_user=vagrant ansible_ssh_pass=vagrant

You’ll recognize the servers we described in the Vagrantfile before. If you don’t provide the ansible_host key-value-pair, a DNS name resolution will be attempted. As Ansible is agentless and uses plain SSH to access the targets, I used the default credentials Vagrant creates when starting the servers. The servers got separated into groups (or *.ini file sections). You can have groups of groups too, which is is a powerful concept. These groups come in handy when applying deployment logic based on these groups. The next section will show that.

Ansible playbook

Ansible uses playbooks to encapsulate deployment and server configuration logic. One playbook can contain 1 to N plays. One play uses a group of servers or a single server as a target. One play contains 1 to N tasks. A task is the atomic building block and smallest unit of work in Ansible.

There is one magic group called all. This includes all the servers in the target file we specified before. The syntax in playbooks is YAML and the file we use is the playbook.yml. I used one playbook to contain all necessary plays to keep it simple. Let’s go through the plays piece by piece. I’ll show the code first and explain it below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

---

# ===========================================================================
# Do basic setup on all hosts
# ===========================================================================
- hosts: all
  become: true
  gather_facts: false

  tasks:
    - name: "Wait for SSH to be ready."
      become: false
      delegate_to: localhost
      wait_for:
        port: 22
        host: '{{ ansible_host }}'
        search_regex: "OpenSSH"
        delay: 5
        timeout: 300

    - name: "Add our servers to the hosts file."
      lineinfile:
        dest: /etc/hosts
        # use the IP address we specified in the Vagrantfile
        line: '{{ hostvars[item].ansible_host }} {{item}}'
      with_items: '{{ groups["all"] }}'

    - name: "Ping each other via DNS names."
      ping:
      with_items: '{{ groups["all"] }}'

    - name: "Ensure system package cache is updated."
      apt:
        update_cache: "yes"
        cache_valid_time: 3600

This is the first play and we use the magic group all as a target. As described before, that includes all servers. From reading the name lines, you should get an idea what happens. You also see that I re-use existing Ansible modules, namely wait_for, lineinfile, ping and apt. I won’t go into the details of the modules I used. A full list of modules is available at [2], take a look at them for the details. The goal of this play is to have the servers ready for deploying the monitoring and applications later.

Keep in mind that everything here happens inside the servers (if not delegated, like I did here to wait for SSH). When working with virtual machines which got created a few seconds earlier, sometimes the SSH service is not fully ready when starting the playbooks, that’s why I added a waiting logic. Prometheus will later use DNS name resolution, that’s why I manipulated the hosts file on these servers.

38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

# ===========================================================================
# Do setup on all hosts we want to monitor
# ===========================================================================
- hosts: all  # we want the metrics of the monitoring server too
  become: true

  tasks:
    - name: "Install Prometheus Node Exporter package."
      apt:
        name: prometheus-node-exporter

    - name: "Ensure the Node Exporter is started and starts at host boot."
      service:
        name: prometheus-node-exporter
        enabled: true
        state: started

    - name: "Check if the service emits metrics."
      uri:
        url: http://127.0.0.1:9100/metrics
        method: GET
        status_code: 200

After we have prepared the servers with the basic steps, we install the operating system packaged version of one of many Prometheus exporters [3], the Prometheus-Node-Exporter. This exporter emits the metrics we are interested in. The Prometheus service will later collect the metrics from this URI.

I like to add small “assert tasks” which check conditions I expect to be there, to fail fast if things go wrong. Here I do a simple HTTP GET request to see if the exporter emits metrics.

Now that we have something to listen on, let’s install the rest of the monitoring.

 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165

# ===========================================================================
# Do prometheus server specific setup only on the monitoring server
# ===========================================================================
- hosts: monitoring
  become: true


  tasks:
    # --- Prometheus --------------------------------------------------------
    - name: "Install the Prometheus server."
      apt:
        name: prometheus

    - name: "Configure the Prometheus server."
      copy:
        src: prometheus.yml
        dest: /etc/prometheus/prometheus.yml
      notify: event_restart_prometheus

    - name: "Ensure Prometheus is started and starts at host boot."
      service:
        name: prometheus
        enabled: true
        state: started

    - name: "Check if Prometheus is accessible."
      uri:
        url: http://127.0.0.1:9090/graph
        method: GET
        status_code: 200

    # --- Grafana -----------------------------------------------------------
    - name: "Install the Grafana server."
      apt:
        name: grafana

    - name: "Copy Grafana configuration file."
      copy:
        src: grafana.ini
        dest: /etc/grafana/grafana.ini
      notify: event_restart_grafana

    - name: "Ensure Grafana is started and starts at host boot."
      service:
        name: grafana
        enabled: true
        state: started

    - name: "Check if Grafana is accessible."
      uri:
        url: http://127.0.0.1:3000
        method: GET
        status_code: 200

    - name: "Add Prometheus as datasource to Grafana."
      vars:
        prometheus_datasource:
          name: "prometheus"
          type: "prometheus"
          url: "http://127.0.0.1:9090"
          access: "proxy"
          isDefault: true
          basicAuth: false
      uri:
        url: http://127.0.0.1:3000/api/datasources
        method: POST
        body: "{{ prometheus_datasource | to_json }}"
        body_format: json
        user: admin
        password: admin
        force_basic_auth: yes
        status_code: 200,500  # 500 means, the datasource is already added
        headers:
          Content-Type: "application/json"
          Accept: "application/json"

    - name: "Upload the example Grafana dashboard."
      uri:
        url: http://127.0.0.1:3000/api/dashboards/db
        method: POST
        body: "{{ lookup('file', 'infra-node-metrics.json') }}"
        body_format: json
        user: admin
        password: admin
        force_basic_auth: yes
        status_code: 200
        headers:
          Content-Type: "application/json"
          Accept: "application/json"


  # --- After all tasks are executed (if notified) --------------------------
  handlers:
    - name: "Restart the Prometheus service."
      service:
        name: prometheus
        state: restarted
      listen: event_restart_prometheus

    - name: "Restart the Grafana service."
      service:
        name: grafana
        state: restarted
      listen: event_restart_grafana

This is a big one. The important parts are highlighted. We apply this logic only on the monitoring server. Three important files get used here:

• prometheus.yml configures Prometheus
• grafana.ini configures Grafana
• infra-node-metrics.json example dashboard

If you want to have a reproducible infrastructure, it’s good to save such things in your version control system too. For the dashboard, I usually create one in the Grafana web UI and use the export function to store the generated JSON. Only for very small changes I edit the JSON file itself.

There are also again some tasks which assert that the services are up and running. The handlers at the end get fired after the tasks are finished and if a notification was triggered.

With this logic, we have the monitoring in place. But we need something to have impact on our resources. We need applications:

168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183

# ===========================================================================
# Push the "applications" to the application servers
# ===========================================================================
- hosts: application-servers
  become: true
  gather_facts: false

  tasks:
     - name: "Copy the applications to the servers."
       copy:
         src: "{{ item }}"
         dest: "/root/{{ item }}"
       with_items:
         - eat_cpu.py
         - eat_disk.py
         - eat_memory.py

A very simple and short one this time. The application code we copy here is shown in Appendix.

Now execute the playbook locally:

1

$ ansible-playbook -i hosts.ini playbook.yml

While this command does its magic, let’s have a look at the configuration files we have copied to the monitoring node.

Prometheus

You have seen in the playbook before, that we copy a file called prometheus.yml. This is what it does:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# The full config is at:
#   https://prometheus.io/docs/operating/configuration/

global:
  # How frequently to scrape targets by default.
  scrape_interval: 15s

scrape_configs:

  - job_name: 'monitoring'
    target_groups:
      - targets: ['localhost:9100']
        labels:
          name: 'monitoring'
          owner: 'mzio'

  - job_name: 'app-server-1'
    target_groups:
      - targets: ['app-server-1:9100']
        labels:
          name: 'app-server-1'
          arch: 'x86'

  - job_name: 'app-server-2'
    target_groups:
      - targets: ['app-server-2:9100']
        labels:
          name: 'app-server-2'
          arch: 'x86'

This is a static configuration, which only makes sense if your environment does not change that often. There are more dynamic ones with service discovery, but I won’t dive into that right now. The highlighted lines are the interesting ones. The meaning of these lines piece by piece:

• scrape_configs: The act of collecting (pulling) metrics from a source is called scraping in Prometheus terms. We can have N scrape configurations.
• job_name: A job abstracts 1 to N targets. For target specific resources (like in this post), this seems unnecessary. Imagine HTTP response times of your distributed, highly available application on the other hand. Then you don’t care about a single HTTP server, but in the combined metrics. A job name could then be “web-ui-app-x” with multiple targets. I plan to write a post about HAProxy at some point, it will make more sense then.
• targets: A scrape job can have multiple targets. We could have added both application servers here, but then both would get the same labels applied. The labels are one of the nice things of Prometheus which distinguishes it from other monitoring software like statsd.
• name: This is simply an arbitrarily chosen free-form label. Labels give you the ability to tag / mark / annotate your metrics. These values can later get used to set constraints in the Prometheus query language.

The best metrics don’t help, if you can’t pull knowledge out of them and derive actions from that knowledge. Visualizing data is the best method (for me) to create knowledge from data, and Grafana does a very good job at data visualization.

Grafana

This is the last file we discuss, before finally monitoring our environment and it’s a very simple one:

1
2
3
4
5
6
7

# Full docs at http://docs.grafana.org/installation/configuration/
# We only keep the changes from the defaults here.

#################################### Anonymous Auth ##########################
[auth.anonymous]
enabled = true
org_role = Viewer

The only thing I change from the defaults is, that I’d like to have reading access for people not logged into Grafana. You still need to be logged in to create, change or delete dashboards.

That’s the last part of the automation we use. Let’s check what we can do with that.