Elastic Stack (formerly ELK) - Kibana (part 1)

Intro

This is the fourth part of a series about the Elastic Stack. As said at the beginning Kibana [1] is on top of Elasticsearch and Logstash. To get the most out of this post, it’s advisable to read the previous posts in this series:

1. Elastic Stack (formerly ELK) - Elasticsearch
2. Elastic Stack (formerly ELK) - Logstash (Part 1)
3. Elastic Stack (formerly ELK) - Logstash (Part 2)

If you already know that content, here we go.

A short recap

The environment I’ve build for this post, consists of 3 virtual machines:

• es1 as host of the Elasticsearch service
• ls1 as host of the Logstash service and the example application
• kb1 as host of the Kibana service

The example application looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#!/usr/bin/env python

import logging
logging.basicConfig(format='%(asctime)s %(levelname)s '
                           '%(process)d %(thread)d '
                           '%(name)s %(funcName)s %(lineno)s '
                           '%(message)s',
                    level=logging.DEBUG,
                    filename="/var/log/example-app/example.log")
logger = logging.getLogger('example')


def do_something():
    logger.debug('I did something!')


def main():
    logger.info('Started the application.')
    do_something()


if __name__ == '__main__':
    main()

The log entries created by this example app will be processed by Logstash and dissected into smaller parts by this pipeline configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

input {
  file {
    id => "example-in"
    path => "/var/log/example-app/example.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  dissect {
    mapping => {
      "message" => "%{time} %{+time} %{level} %{pid} %{thread} %{loggername} %{function} %{line} %{msg}"
    }
    remove_field => [ "message" ]
    convert_datatype => {
      line => "int"
      pid => "int"
      thread => "int"
    }
  }
  date {
    match => [ "time", "ISO8601" ]
    remove_field => [ "time" ]
  }
}

output {
  elasticsearch {
    id => "example-out"
    hosts => ["http://es1:9200"]
    index => "example"
  }
}

The Elasticsearch Index example here is the one we will use in Kibana later.

Set up the environment

To reproduce the steps in this post, you need to have installed locally:

• Vagrant [2]
• VirtualBox [3]

After these prerequisites are fulfilled:

1. download the compressed project source files.
2. extract the archive
3. change to the env directory
4. start the Vagrant setup

1
2
3
4

$ wget http://www.markusz.io/_downloads/elastic-stack-elk-kibana-part1.tar.gz
$ tar -zxvf elastic-stack-elk-kibana-part1.tar.gz
$ cd env
$ vagrant up  # does also all of the installation

After this is fully done, all services are up and running, and the example application got executed once, so that we have some data to query.

Set up the index

Open your browser and point it to http://192.168.73.13:5601 and you should be greeted with this:

Click the button Set up index patterns in the top right corner, to make Kibana aware of the index example we created in Elasticsearch during the setup of the environment.

Enter example in the text field below Index pattern:

After a click on Next step you need to chose the @timestamp filed for Time Filter field name:

After a click on Create index pattern you have the option to do some fine tuning:

We will ignore that in this post. It’s worth mentioning though, that the data types we specified in the Logstash pipeline before gets recognized here. See at the field line for example, which is a number and not a string.

Discover your data

After the index is setup, we can start to query our logs. Click on Discover on the left panel and adjust the time range on the top right to Today:

Go to the left hand side, below Available Fields and click on the add button which appears when you hover with your cursor above the fields:

1. host
2. level
3. loggername
4. function
5. msg
6. line
7. pid

Let’s use the Lucene query syntax [4] to do some searches.

Enter level:info pid:2523 into the search bar and hit the search button and you will see that these terms get joined with OR:

Add the plus symbol in front of the field name to specify that it must be in the result set. Enter +level:info pid:2523 and search again:

Wildcard searches are possible too. Use the query +host:ls* +function:do_something to search for logs of all hosts which start with the name ls:

A search with ranges can also be done. Search with line:[10 TO 15] for all log entries which are created in the code between lines 10 to 15:

These are fairly simple examples to help understand the basic idea behind it. A more realistic data source to practice with can be found at [5] which contains all logs from the continuous integration tooling of the OpenStack community.

Summary and outlook

This post showed how to set up Kibana on an existing Elasticsearch data store to search for log entries an example application was producing. A few example queries with the Lucene search query showed how you can build search terms to find the log entries most important to you. The information how to setup these services can be found in the project source files.

What I didn’t show here, and which will be the content of part 2 in a follow-up post, are:

• ways to save your queries,
• create visualizations and dashboards in the UI
• store them in your code repository of choice.

References