August 16, 2017
HomeComputer networksIcinga 2 with InfluxDB & Grafana on CentOS 7

Icinga 2 with InfluxDB & Grafana on CentOS 7

By  Computer networks, Information security, Monitoring  0 Comments

Foreword – PHP-FPM

Icinga now uses PHP-FPM.  Part of this installation guide needs an update to reflect that, I’ll cover it later on when my current home projects are done.

Structure of the article

This article is less structured than other tutorials I’ve written on this website.  It is more a step-by-step howto, often skipping the detailed explanation.  In summary, it explains how to:

  • Install Icinga2, mostly following the official documentation
  • Setting up Icinga Web, including SSL
  • Historical data storage and graphics with InfluxDB and Grafana, with integration in Icinga Web 2

Important notes:

  • The keyword “icinga” is used as the password in this article.  Of course, for production/pentesting purposes, you should use a real password.
  • This setup supposes that you have a DNS server up and running in your network.  Without it, the Apache vHost strategy will not work.
  • This tutorial supposes that Icinga2, InfluxDB & Grafana are all running on the same host.  If this is not the case, the IP addresses obviously need to be adapted.

Icinga 2

Icinga packages

First install Icinga core packages:

# yum install https://packages.icinga.org/epel/7/release/noarch/icinga-rpm-release-7-1.el7.centos.noarch.rpm
# yum install icinga2 vim-icinga2 icinga2-selinux epel-release sudo
# systemctl enable icinga2
# systemctl start icinga2
# yum install nagios-plugins-all nagios-plugins-nrpe

Database setup

Monitoring with Icinga creates a lot of data.  PostgreSQL has the edge over MySQL/MariaDB in that kind of situations.

# yum install postgresql-server postgresql icinga2-ido-pgsql
# postgresql-setup initdb
# systemctl enable postgresql
# systemctl start postgresql

Separate virtual disk on /monitoring to store the Icinga data.

# mkdir /monitoring/var-lib-pgsql
# systemctl stop postgresql
# mv /var/lib/pgsql/* /monitoring/var-lib-pgsql
# chcon -Rv --type=postgresql_db_t /monitoring/var-lib-pgsql
# echo "/monitoring/var-lib-pgsql /var/lib/pgsql none bind 0 0 >> /etc/fstab
# mount -a
# systemctl start postgresql

When playing with SELinux it is better to usz bind mount rather than symlinks.  The bind mount is invisible to SELinux, whereas the symlink is a specific file type that requires dedicated policies.

Now, create the database:

# cd /tmp
# sudo -u postgres psql -c "CREATE ROLE icinga WITH LOGIN PASSWORD 'icinga'"
# sudo -u postgres createdb -O icinga -E UTF8 icinga
# sudo -u postgres createlang plpgsql icinga
# sudo -u postgres createdb -O icinga -E UTF8 --lc-collate=en_US.utf8 --lc-ctype=en_US.utf8 -T template0 icinga

Modify the PostgreSQL configuration to allow the icinga user with md5 authentication locally.  In order for everything, including the web interface, to work, the /var/lib/pgsql/data/pg_hba.conf file should be:

# icinga
local   icinga      icinga                            md5
host    icinga      icinga      127.0.0.1/32          md5
host    icinga      icinga      ::1/128               md5
# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
host    all             all             192.168.11.0/24         md5
host    all             all             10.8.0.0/24             md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

Restart PostgreSQL:

# systemctl restart postgresql

After creating the database and permissions you can import the Icinga 2 IDO schema using the following command:

# export PGPASSWORD=icinga
# psql -U icinga -d icinga < /usr/share/icinga2-ido-pgsql/schema/pgsql.sql

Enabling the IDO PostgreSQL module

Next, the IDO PostgreSQL module of Icinga must be enabled.  Uncomment the lines in /etc/icinga2/features-available/ido-pgsql.conf.  Enable the ido-pgsql feature configuration file using icinga2 feature enable:

# icinga2 feature enable ido-pgsql
Module 'ido-pgsql' was enabled.
Make sure to restart Icinga 2 for these changes to take effect.
# systemctl restart icinga2

Moving logs

On top of DB elements, the Icinga2 log can get crowded too.  We’ll use the same strategy with the /var/log/icinga2 folder.

# mkdir /monitoring/var-log-icinga2
# systemctl stop icinga2
# mv /var/log/icinga2/* /monitoring/var-log-icinga2/
# chcon -Rv --type=nagios_log_t /monitoring/var-log-icinga2
# echo "/monitoring/var-log-icinga2 /var/log/icinga2 none bind 0 0 >> /etc/fstab
# mount -a
# systemctl start icinga2

External command pipe

We’re not setting up the web interface immediately, but we need it to be installed to setup the command pipe.  Web interfaces and other Icinga addons are able to send commands to Icinga 2 through it.

# yum install httpd
# systemctl enable httpd
# systemctl start httpd
# icinga2 feature enable command
# systemctl restart icinga2

By default the command pipe file is owned by the group icingacmd with read/write permissions. Add your webserver’s user to the group icingacmd to enable sending commands to Icinga 2 through your web interface:

# usermod -a -G icingacmd apache

You can verify that the user has been successfully added to the icingacmd group using the id command:

$ id apache

We’re done for Icinga 2 basic setup, but right now you only have a monitoring server set up without much working.  Before configuring hosts to be monitored, we’ll setup the web interface.

Icinga Web 2

Packages installation

# yum install icingaweb2 icingacli php-pgsql icingaweb2-selinux
# usermod -a -G icingaweb2 apache

Modify /etc/php.ini to fix the TimeZone, e.g. to:

[Date]
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
date.timezone = Europe/Brussels

Then restart Apache to take the change into account and create the icingaweb database:

# systemctl restart httpd
# sudo -u postgres createdb -O icinga -E UTF8 icingaweb

phpPgAdmin

Unavoidably, something will go wrong at some point and you’ll need to take a peek at the status of your configuration.   Install phpPgAdmin following [4].

Configuration of the web interface

You can set up Icinga Web 2 quickly and easily with the Icinga Web 2 setup wizard which is available the first time you visit Icinga Web 2 in your browser. When using the web setup you are required to authenticate using a token. In order to generate a token use the icingacli:

# icingacli setup token create

Visit Icinga Web 2 in your browser to access the setup wizard and complete the installation: /icingaweb2/setup.

Once you’re done, tadaa, you can login on Icinga 2 web interface.

Configuration files

The Icinga2 configuration highly depends on your infrastructure.  Follow the documentation, it’s very well written and full of examples.

I often end up with permission issues on the notification scripts.  I ended up copying the notification scripts to another folder:

# mkdir /monitoring/notification-scripts
# cp /etc/icinga2/scripts/* /monitoring/notification-scripts/
# cd /monitoring/notification-scripts
# chmod a+x *
# chown icinga:icingacmd *
# chcon -v --type=nagios_notification_plugin_exec_t *

The notification path scripts can be changed, if you keep the default folder structure, in /etc/icinga2/conf.d/commands.conf.

Time-series database and graphs

To keep history of the perfdata and displays graphs in the web interface, we will use the InfluxDB + Grafana couple.  InfluxDB is a time-series database that will store data, whereas Grafana will dynamically generate graphs that can be integrated in the web interface.

Install InfluxDB & Grafana

This section is loosely based on [5].  I simply didn’t follow the Grafana dashboards given there, as I preferred to base my work on the now actively developed Grafana module for the Icinga 2 web interface.

InfluxDB

Define the repository in the yum configuration, then install and enable influxdb:

# cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF
# yum install influxdb
# systemctl enable influxdb
# systemctl start influxdb

Create an icinga2 database & user in influxdb:

# influx
create database icinga2;
create user icinga2 with password 'icinga';
grant all on icinga2 to icinga2;
quit;

We now have an InfluxDB database ready to receive data from Icinga.  The next step is to configure the writer on the Icinga side.  First, enable the feature:

# icinga2 feature enable influxdb

Then, configure the writer in /etc/icinga2/features-enabled/influxdb.conf:

/**
  * The InfluxdbWriter type writes check result metrics and
  * performance data to an InfluxDB HTTP API
  */

library "perfdata"

object InfluxdbWriter "influxdb" {
  host = "127.0.0.1"
  port = 8086
  database = "icinga2"
  username = "icinga"
  password = "icinga"
  host_template = {
    measurement = "$host.check_command$"
    tags = {
      hostname = "$host.name$"
    }
  }
  service_template = {
    measurement = "$service.check_command$"
    tags = {
      hostname = "$host.name$"
      service = "$service.name$"
    }
  }
  enable_send_thresholds = true
  enable_send_metadata = true
}

Now, restart Icinga2 to have the InfluxDB writer running:

# systemctl restart icinga2
Grafana

Define the repository, install Grafana and enable the service:

# cat <<EOF | sudo tee /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/7/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF
# yum install -y grafana
# systemctl enable grafana-server
# systemctl start grafana-server

The Grafana web interface lives on port TCP:3000.  Login with the default credentials (admin:admin)  and create a new datasource:

I finalized the installation following [6].  It all works like a charm with no work!

One last important word: if you use the nrpe_local.cfg file, make sure to change its SELinux context:

# chcon -Rv --target=nrpe_etc_t nrpe_local.cfg

References

Related Posts

About Author

Eric Viseur