Configuring monitoring services using Icinga

Introduction

This page describes how Alexis Huxley installed Icinga.

IMHO, monitoring facilities should be as independent of all other systems and services as possible in order that they should be as reliable as possible. For this reason Icinga authentication will be done by Apache. Alternatives include using LDAP and database backends, but these are not discussed here.

alert I had used Icinga 1 and found it very intuitive, but with Icinga 2 I had several problems:

  1. no documentation for server-client configurations (only available documention I could find was for server-satellite configurations)
  2. unofficial documentation suggested, when adding a client, to run: 
    trenne# icinga2 node list
Warning: CLI command 'node list' is DEPRECATED! Please read the Changelog.
...
trenne#

    However, no Changelog files were included and a search of the official documention did not explain what superceded this command

  3. many references in the packages to repository.d which seems to be also obsolete.

At this point I decided that I had spent too much time trying to configure Icinga 2 and switched to Zabbix. This page remains here in case I ever decide to try again.

Installing Icinga

  1. Configure access to the official Icinga repository by either running: 
    add-apt-repository ppa:formorer/icinga

    or by running:

    apt-key adv --quiet --keyserver pgp.mit.edu --recv-keys 36862847

    and adding the following to /etc/apt/sources.list:

    http://ppa.launchpad.net/formorer/icinga/ubuntu xenial main
  2. Install the icinga server with: 
    apt-get update
apt-get install icinga2
  3. Verify basic installation by running the following command and comparing the output: 
    icinga2 feature list
Disabled features: api command compatlog debuglog gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: checker mainlog notification
systemctl status icinga2
● icinga2.service - LSB: icinga2 host/service/network monitoring and management system
   Loaded: loaded (/etc/init.d/icinga2; bad; vendor preset: enabled)
   Active: active (running) since Fri 2016-09-23 14:24:45 CEST; 17min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/icinga2.service
           └─22411 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon -d -e /var/log/icinga2/icinga2.err

Sep 23 14:24:45 trenne systemd[1]: Starting LSB: icinga2 host/service/network monitoring and management system...
Sep 23 14:24:45 trenne icinga2[22355]:  * checking Icinga2 configuration
Sep 23 14:24:45 trenne icinga2[22355]:    ...done.
Sep 23 14:24:45 trenne icinga2[22355]:  * Starting icinga2 monitoring daemon icinga2
Sep 23 14:24:45 trenne icinga2[22355]:    ...done.
Sep 23 14:24:45 trenne systemd[1]: Started LSB: icinga2 host/service/network monitoring and management system.
  4. Install plugins 
    apt-get install nagios-plugins
  5. To support syntax highlighting in vim, run: 
    apt-get install vim-icinga2 vim-addon-manager
vim-addon-manager -w install icinga
  6. Configure the Icinga2 web interface according to the following sub-procedure:
    1. Run: 
      apt-get install mariadb-client mariadb-server
apt-get install icinga2-ido-mysql    #  this is slow
icinga2 feature enable ido-mysql
systemctl restart icinga2
apt-get install apache2
#  See https://github.com/Icinga/icingaweb2/blob/master/doc/02-Installation.md regarding next line
echo 'ModPagespeedDisallow "*/icingaweb2/*"' > /etc/apache2/conf.d/icinga2
icinga2 feature enable command
systemctl restart icinga2
usermod -a -G nagios www-data
    2. Edit /etc/php/7.0/apache2/php.ini and set: 
      date.timezone = "Europe/Berlin

      or your own timezone. You can find a list of timezones at https://www.php.net/manual/en/timezones.europe.php or you can run the following command to find the timezones available on your system:

      cd /usr/share/zoneinfo
find */ -type f
    3. and then restart apache so it sees the changed setting: 
      systemctl restart apache2
    4. Later, when we are asked how users are to authenticate themselves with the Icinga web interface, then we will choose to make Apache authenticate users and to propagate those credentials on to Icinga. Some preparation is needed for this:
      1. Run: 
        a2enmod auth_digest
systemctl restart apache2
touch /etc/apache2/icinga.digest
htdigest /etc/apach2/icinga.htdigest "Icinga Service" icingaadmin  #  set password
      2. Edit /etc/apache2/conf-enabled/icingaweb2.conf and add the following in the specified context: 
        <Directory "/usr/share/icingaweb2/public">
    Require valid-user
        AuthType Digest
        AuthName "Icinga Service"
        AuthUserFile /etc/apache2/icinga.htdigest
    ...
</Directory>
    5. Run the following command to generate a token (a temporary password), which will be needed to continue configuring Icingaweb2: 
      icingacli setup token create
    6. Go to http://<server>/icingaweb2/setup, check this requires authentication as set up above and log in as icingadmin.
    7. accept the default list of enabled modules and click Next.
    8. check that the only modules not green are for database backends that you are not using and click Next.
    9. set authentication type to external (meaning that Apache will authenticate and pass credentials to Icinga), check there is no warning displayed (if you arrived at the page without authenticating then there will be a warning) and click Next.
    10. Click ‘Validate configuration’ without entering a filter pattern and if everything is okay then click Next.
    11. At the ‘Administration’ page, the username ‘icingaadmin’ should already be filled in, so just click Next.
    12. At the ‘Application administration’ page, accept all defaults and click Next.
    13. At the summary page, just click ‘Next’.
    14. At the ‘Welcome to the configuration of the monitoring module …’ page, click Next.
    15. At the ‘Monitoring backend’ page, accept the defaults and click Next.
    16. At the ‘Monitoring IDO Resource’ page, enter the Icinga database details, which can be seen in the file /etc/icinga2/features-available/ido-mysql.conf, click ‘Validate Configuration’ and if everything is okay then click Next.
    17. At the ‘Command Transport’ page, accept the defaults and click Next.
    18. At the ‘Monitoring Security’ page, accept the defaults and click Next.
    19. At the summary page, just click ‘Finished’.
    20. At the ‘Congratulations …’ page, click ‘Log in to Icinga Web 2’.
  7. Communication between the Icinga server and the monitored hosts uses certificates for authentication. Set this up as follows:
    1. Run: 
      icinga2 node wizard

      and answering ‘n’ to the first question (in order to make a master node configuration) and accept all other defaults.

    2. Run: 
      systemctl restart icinga2
  8. If desired, proxy Icinga through a front-end webserver by adding the following the the front-end webserver’s configuration: 
    ProxyPass /icingaweb2/ http://<backend-cinga-server>/icingaweb2/
ProxyPassReverse /icingaweb2/ http://<backend-icinga-server>/icingaweb2/

    and run:

    service apache2 reload

    But note that this currently requires the authentication to be done on the backend Icinga server not the front proxy.

Notes on configuration

According to the official documentation, use of ‘icinga2 repository’ commands is deprecated.

Adding users

  1. Add the user to /etc/apache2/icinga.htdigest. That’s it!

Removing users

Adding clients

In Nagios and earlier versions of Icinga, entirely different packages were installed on the monitoring server and the monitored hosts. In Icinga2 different features are enabled on the monitoring server and the monitored hosts, but the same core packages installed on both.

  1. Configure access to the official Icinga repository by either running: 
    add-apt-repository ppa:formorer/icinga

    or by running:

    apt-key adv --quiet --keyserver pgp.mit.edu --recv-keys 36862847

    and adding the following to /etc/apt/sources.list:

    http://ppa.launchpad.net/formorer/icinga/ubuntu xenial main
  2. Install the icinga server with: 
    apt-get update
apt-get install icinga2 monitoring-plugins
  3. Run: 
    icinga2 node wizard

    and follow the prompts, which will include answering ‘Y’ to create a satellite node, specifying the name of the Icinga server and copy-and-pasting a command to the Icinga server.

  4. To work around a bug run: 
    mkdir /var/log/icinga2
chown nagios:adm /var/log/icinga2
chmod 2751 /var/log/icinga2
  5. Run: 
    systemctl restart icinga2
  6. On the server run: 
    icinga2 node list
  7. to verify the new host is known of and update the server configuration with: 
    icinga2 node update-config
systemctl restart icinga2

Note that I had problems with log dirs being removed, which caused the service to fail to start. It was (temporarily) fixed with:

farfalle# mkdir /var/log/icinga2
farfalle# systemctl start icinga2.service
farfalle#

Removing clients

Adding services

Removing services

  1. Disable ping6 checks as follows:
    1. Run: 
      icinga2 object list --type Host

      and verify that presence of address6 variables.

    2. Comment out the address6 assignment in /etc//icinga2/conf.d/hosts.conf.
    3. Reload the configuration with: 
      service icinga2 reload
    4. Again run: 
      icinga2 object list --type Host

      and verify that absence of address6 variables.

Hopefully obsolete documentation about a previous version of Icinga to be removed soon

  1. and run: 
    service apache2 reload

    and copy /etc/apache2/icinga.htdigest from the back-end webserver to the front-end webserver.

  2. In order to allow the nagios user to install commands on remote hosts,
    1. Change the nagios user’s shell to /bin/bash (in order to allow ‘make’ to install commands as that user) by running: 
      chsh --shell=/bin/bash nagios
mkdir -p /usr/local/opt/nagios
chown nagios:nagios /usr/local/opt/nagios
    2. If this is an upgrade, then copy over the nagios user’s SSH key pair and known_hosts file.
  3. If you have your own configuration mechanism then:
    1. Make Icinga take its configuration from where your mechanism will write it. On my own network this means I should edit /etc/icinga/icinga.cfg and change this: 
      cfg_file=/etc/icinga/commands.cfg
# Debian also defaults to using the check commands defined by the debian
# nagios-plugins package
cfg_dir=/etc/nagios-plugins/config

      to this:

      #cfg_file=/etc/icinga/commands.cfg
# Debian also defaults to using the check commands defined by the debian
# nagios-plugins package
#cfg_dir=/etc/nagios-plugins/config
    2. Install your own configuration mechanism. On my own network this means I should run: 
      su -
mkdir ~/opt
svn co https://svn.pasta.freemyip.com/main/nagiostools/trunk nagiostools
make -C icinga-objects
make -C custom-check-commands
exit
  4. If hosts are frequently reinstalled, then you may wish to run the following: 
    su - nagios
mkdir -p .ssh
echo -e 'Host *n    UserKnownHostsFile /dev/nulln    StrictHostKeyChecking non    LogLevel ERROR' >> config
exit

    (The first two directives add the remote host key automatically to a file that contains no conflicts, and the second suppresses the message about the key being added.)

Adding clients

  1. Run: 
    apt-get install nagios-plugins
  2. If there is not a ‘nagios’ account then run: 
    groupadd --system nagios
useradd --system --home-dir /var/local/nagios --gid nagios --create-home nagios
  3. If nagios’s shell is not /bin/bash then run: 
    chsh --shell=/bin/bash nagios
  4. Run: 
    TRUSTED_KEY="<trusted-ssh-key>"
mkdir -p ~nagios/.ssh
echo "$TRUSTED_KEY" >> ~nagios/.ssh/authorized_keys
chown -R nagios:nagios ~nagios/

    and then the get nagios@<icinga-server> to cache the system’s host SSH key.

  5. Create a directory for custom checks by running: 
    mkdir -p /usr/local/opt/nagios
chown -R nagios:nagios /usr/local/opt/nagios
  6. Install any custom checks.
  7. Some checks may require sudo rights. For my own systems I need to replace /etc/sudoers’s contents with the following: 
    nagios ALL = NOPASSWD: /usr/local/opt/nagios/plugins/check_fs_writable
  8. On the Icinga server update the configuration accordingly (see the previous section for how to do this).

See also