As an it slave, I have the same problems as most datacenters:

• Running out of space because of more and more machines get into the datacenter
• Overheating
• Powerconsumption

Added to the "normal" datacenter problems, I also have to deal with:

• Wife acceptance
• Noice, my neighbours has complained about the noice outside my basement.

To address this issues I will run a consolidation project to get fewer machines by using virtualization. As virtualization engine, KVM is choosed.

This article will describe some pitfalls I run into and how I solved them.

KVM plattform

KVM seems to be the future for virtualization within the Linux area since Xen has done some mistakes.

As I have good experience of running CentOS and has KVM included I decided to use CentOS as the plattform for my virtualization.

Issues

Bridge network

To let the virtual boxes has full access to the network you need create a network bridge, it is rather straightforward and well documented so I will skip this part. I used the documentation on Red Hat customer support. Why KVM requires a bridge instead of a normal NIC is another question

Snapshots

One handy feature with virtualization is that the virtual systems can be one big fat file at the physical host, this gives the possibility to make full backups without turning the system down a.k.a snapshots.

However to get snapshot to work the filetype must be qcow2 and raw is the default format.

It is possible to convert from raw to qcow2 format by running the command:

qemu-img convert -f raw -O qcow2 <virtualhost>.img <virtualhost>.img.qcow2

Remember to turn of your virtual system by running:

virsh shutdown <virtualhost>

After the conversion:

virsh edit <virtualhost>

modify the type and path, i.e.

      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/web.img.qcow2'/>

Start your virtual system:

virsh start <virtualhost>

Take a snapshot with:

qemu-img snapshot -c <snapshotname> <virtualhost>.img.preallc.qcow2

To create a full image of your snapshot you need to compile a more resent version of qemu-img then shipped with CentOS 6.2, I download a later version and compiled it from here, i renamed the new to qemu.img2 and copied it to /usr/local/bin

qemu-img2 convert -p -f qcow2 -O qcow2 -s <snapshotname> <virtualhost>.img.preallc.qcow2 <targetpath>

Beware: This command can use all your resources and may affect all your virtual systems, use nice and ionice to prevent it.

Disk I/O

After installing just a few virtual system I noticed that my physical host got alot of I/O wait, it can be seen by using tools like top.

To monitor this I installed the nagios plugins:

• check disk io, can be found here
• check cpu stats, can be found here
• check_libvirt, developed by op5 and an Howto

However it seems like that though my virtual systems did more or less nothing they caused alot of disk I/O on the physical system.

I did some investigations and read quite many articles, fiddled and tested. The following is what worked for me in my setup.

#1 Change disk scheduler

The disk scheduler can be changed on runtime by modify the file:

/sys/block/sda/queue/scheduler

to see what scheduler you use now:

cat /sys/block/sda/queue/scheduler

Change sda to the device you have.

The scheduler that worked best for me is deadline

echo deadline > /sys/block/sda/queue/scheduler

#2 Mount with noatime

A feature in Unix and other Unixlike systems like Linux is that it normaly stores when a file is accessed. So one read always produce a write and if you are using raid like morroring this get worse, one reade always generate several writes. This creates alot of overhead for a feature soldom used.

So change /etc/fstab so it will mount the file systems with noatime.
An axample:

UUID=a290aa4b-635c-45fa-b144-1fbef90b3735 /                       ext4    defaults,noatime        1 1

#3 Preallocation disk images

A real boost that is hidden in the featureset and not shown in the virtualmachine gui is using preallocation disk images. They cannot be created from the GUI so I install the virtual machine, turn it of and convert it afterwards.

qemu-img convert -f qcow2 -O qcow2 -o preallocation=metadata <virtualhost>.img.qcow2 <virtualhost>.img.preallc.qcow2

Change the path to new image name by editing the virtual machine settings

virsh edit <virtualhost>

Conclusions

In my opinion it seems like KVM is still a little bit immature or at least the tools to handle it. Maybe it would be a good idea to have one linux distro focusing on beeing the best platform for virtualization.

I am convinced that I can get even more bang for the bucks out of my installation if I learn how to tweak it even more, so if you have any hints, do not hesitate to contact me. Preferably as a comment to this blogpost.

Referenses:

• op5 Monitor, a nagios based enterprise monitor tool to monitor my environment
• check_libvirt, a nagios plugin done by op5 to monitor KVM
• check_diskio, a nagios plugin to monitor disk I/O
• check_cpu_stats, a nagios plugin to monitor cpu usage rather detailed
• Another blogpost which helped me
• KVM performance improvements and optimizations – Red Hat presentation

As a monitoring fanatic, I think that have mirrored disk without monitor the mirror is worse then not have a redundant disk system at all. If you think that you are safe and you are not, is worse then be aware of a risk and calculate with it.

This article will describe how I did monitoring my FreeNAS system mirrored disks status using Nagios or op5 Monitoring. I assume basic knowledge of Nagios or op5 Monitor and that a Nagios agent is installed.

Demarcation

This article will focus on GEOM raid monitoring. Of cource all important services should be monitored, like HTTP, CIFS, SMB, disk space, CPU Load, memory usage and so on. However that is basic monitoring knowledge and there is alot of documentation on the net howto achive that.

Monitor the GEOM mirror

The most interesting part is to monitor the GEOM mirror to make sure that that the raid1 is working. 

After some googleing I found check_geom, downloaded it and put it in /usr/local/libexec/nagios/ where the other plugins are located.

Add a line to /etc/nrpe.cfg

command[check_geom_mirror]=/usr/local/libexec/nagios/check_geom mirror

and test from Nagios/op5 Monitor host:

# ./check_nrpe -H fnas -c check_geom_mirror
OK mirror  - mirror/2TBmirror0 COMPLETE { ad6 , ad4 }

It works!
 

Configure Nagios or op5 Monitor

In /opt/monitor/etc/services.cfg at the op5 Monitor host

# service 'Disk Raid Mirrors'
define service{
    use                            default-service
    host_name                      fnas
    service_description            Disk Raid Mirrors
    check_command                  check_nrpe!check_geom_mirror
    contact_groups                 it-slav_mail,it-slav_jabber
    }

The result

I tested unplugging the SATA cabel to one of the mirrored disk and after a couple of minutes I got an Critical notification that my disk mirror is broken, I plugged SATA cabel and got a Warning:

After a couple of hours the mirror was OK.

Useful links:

• Previous article describing howto install nagios or op5 Monitor agent on FreeNAS
• op5, an Enterprise Monitor product company
• op5 Monitor, Enterprise Monitoring based on opensource
• Nagios, an excellent opensource monitor tool
• FreeNAS, very capable NAS based on FreeBSD

My old NAS built on top of CentOS has some performence issues and the disks are quit old now. The hardware is leftovers that is approx 8 years old and not highend at that time either. The disks age has worried my for a time and I would like to have higher redunduncy with raid1/mirrored disks.

After some research I decided to run my new NAS on FreeNAS, which has all the features I use, like: CIFS, NFS, FTP, SSH, Rsync, Unison, iSCSI and also easy administration and disk management.

This article will describe how I did setup Nagios or op5 Monitoring on my FreeNAS system. I assume basic knowledge of Nagios or op5 Monitor.

Installing nrpe

Nrpe is the agent that nagios and op5 Monitor uses to monitor a system from the inside.

From the webgui add group nagios and user nagios. User nagios should be a meber of group nagios.

 freenas:~# pkg_add -r nrpe2
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/Latest/nrpe2.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/perl-5.10.1.tbz... Done.
Removing stale symlinks from /usr/bin...
    Skipping /usr/bin/perl
    Skipping /usr/bin/perl5
Done.
Creating various symlinks in /usr/bin...
    Symlinking /usr/local/bin/perl5.10.1 to /usr/bin/perl
    Symlinking /usr/local/bin/perl5.10.1 to /usr/bin/perl5
Done.
cd: can't cd to /usr/include
Cleaning up /etc/make.conf... Done.
Spamming /etc/make.conf... Done.
Cleaning up /etc/manpath.config... Done.
Spamming /etc/manpath.config... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/libiconv-1.13.1_1.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/gettext-0.17_1.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/nagios-plugins-1.4.14,1.tbz... Done.
You already have a "nagios" group, so I will use it.
You already have a "nagios" user, so I will use it.

**********************************************************************
 

freenas:~# cp /usr/local/etc/nrpe.cfg-sample /etc/nrpe.cfg

I changed the allowed_hosts line and added my op5 Monitor host.

So the nagios agent works.

Links

• op5, an Enterprise Monitor product company
• op5 Monitor, Enterprise Monitoring based on opensource
• Nagios, an excellent opensource monitor tool
• FreeNAS, very capable NAS based on FreeBSD

The widget keeps track of how many times the word "moln" appear on idg.se webpage and displays it in a easy view in The Tactical Overview.

Screenshoots from the different modes:

Links:

• op5 Sandbox with a live cloud widget online, registration is necessary
• Ninja, the new and inovative nagios gui
• Widget exchange area

Kudos to op5 support that has created this excellent widget 

It is very handy to have a temprobe attached to network devices like a Wlan router, if OpenWRT is installed it is easy to use 1-wire devices. By using this method it is possible to create wireless temperature probes that uses Wlan.

Install

root@OpenWrt:~# opkg install owserver owfs         
 

Using

root@OpenWrt:~# owserver -u -C --usb_regulartime -p 3001

root@OpenWrt:~# mkdir /mnt/1-wire         


root@OpenWrt:~# owfs -s localhost:3001 -m /mnt/1-wire

root@OpenWrt:/mnt/1-wire/10.87507C010800# ls /mnt/1-wire/
10.87507C010800  81.543A2A000000  alarm            bus.0            settings         simultaneous     statistics       structure        system           uncached

It works!         
 

To make sure that owserver starts when the router is rebooted, add to /etc/rc.local


/usr/bin/owserver -u -C --usb_regulartime -p 3001

Firewall

My op5 Monitor/Nagios system is on the Wan side of the OpeWRT box so I need to open the firewall to port 3001.

Add to /etc/config/firewall

#1-wire ok on Wan
config rule
    option src        wan
    option dest_port    3001
    option target        ACCEPT
    option proto        tcp
 

Result

To gather the data from 1-wire I use op5 Monitor and my Nagios plugin check_1-wiretemp, the result can be seen below

Links

• OpenWrt
• owfs
• op5 Monitor

The dashboard has been modified and improved by the community and an example can be seen at op5 sandbox dashboard and can be downloaded from op5 community.

The original project place is http://dingleberry.me/2010/04/our-new-dashboard/

When I was preparing a presentation about what op5 is doing and our contribution to the community, I went to ideas.nagios.org. When I browsed the list of the biggest issues with Nagios I found out that op5 has packaged and solved them all in op5 Monitor. I encourage everyone to take a peak at the list and judge for them self what platform you want to use for your enterprise monitoring solution.

1. Nagios Clusters

op5 has developed Merlin (Module for Effortless and Redundant and Loadbalanced Infrastructure with Nagios). It provides the possibility to have a redundant or/and loadbalanced solution using Nagios. It also provides a scalable database backend for Ninja and Nagvis, take a peak at Merlin webpage.

2. Performance Graphing

In op5 Monitor op5 has added pnp4nagios that graphs more or less any check that gives a numerical value back.

3. New Status Map

We provide a new statusmap, it is a part of Nagvis project and called Automap. op5 has contributed to Nagvis by adding support for Merlin database backend and GoogleMaps integration

4. Better Interface

op5 has created a new PHP based interface that makes you proud of your monitor solution, called Ninja. Ninja stands for Nagios is now Just Awesome, it looks so good so you can show it to your manager:-) Take a peak at Ninja webpage.

We had a naming competition at op5 my suggestion to call it Yang, Yet Another Nagios Gui, did not get so many votes so the name become Ninja.

5. Configuration

Any one that has tried to configure Nagios by hand has thought bad thoughts. It is cumbersome and in the begining it makes even a skilled Unix admin frustrated. op5 has developed a web based configuration tool named Nacoma. It is included in op5 Monitor.

6. SLA reports

op5 has got alot of feedback from the customers and one issue was better reports, we have created two reports: availability and SLA report. Both of them can be scheduled and looks god compared to the reports in Nagios, they are included in Ninja

 
7. More members in core team

Well, this is something that op5 cannot controll, however, Andreas Ericsson one of op5s core developers are a member og Nagios steering board.

8. SNMP Trap receiver

This is something op5 is working on right now, we have today created a prototype together with a customer and depending of the outcome of that project we will know better when it will be available to the market.

9. UI-Improvment

op5s solution to this issue is Ninja, a better, nicer, scalable, database backend gui to Nagios.

10. Nagios Dashboard

A newTactical Overview with widgets or Nagvis, all included in Ninja

Links

• ideas.nagios.org http://ideas.nagios.org
• Nagios http://www.nagios.org
• op5 http://www.op5.com
• op5 Monitor http://www.op5.com/op5/products/network-monitor
• Ninja http://www.op5.org/community/plugin-inventory/op5-projects/ninja
• Merlin http://www.op5.org/community/plugin-inventory/op5-projects/merlin
• Nagvis http://www.nagvis.org/
• pnp4nagios http://docs.pnp4nagios.org/

With my new and fancy HTC Desire I wanted to look at my Nagios or op5 Monitor status. I have found two apps Nagroid and NagMonDroid.

NagMonDroid did I not get working, probably because op5 Monitor only uses https as protocol.

Nagroid works like a charm:

The configuration is rather straightforward:

Background

It is always a good idea to monitor the server hardware, in many cases the root cause of the probblem is hardware related like: a fan stops and the temperature gets to high, dust in the machine makes it to hot, disks that fails, memory corruption and so on.  This article will describe howto enable hardware monitoring on a HP Proliant running CentOS Linux and then howto collect the data with Nagios or op5 Monitor. The procedure is the same with RedHat Enterprise Linux and similiar with Suse Enterprise server.

The HP manuals and information is bloated with irrelevant information and I had to struggle several hours, ask collegues to get it running. I hope this blog article will help others to get monitoring of HP Proliant using HP Insight Manager easier to setup.

Installing the software on the target system

You need two packages from HP and can be downloaded from hp.com under "Support & Drivers". Search your hardware plattform and correct operating system:

• ProLiant Support Pack for Red Hat Enterprise Linux 5 (i686)  The latest Proliant Support Package 2010-03-02 name is psp-8.25.rhel5.i686.en.tar.gz
• HP System Health Application and Insight Management Agents for Red Hat Enterprise Linux 5 (x86). The latest name is hpasm-8.0.0-173.rhel5.i386.rpm

Install kernel source code and rpm tools:

# yum install kernel-devel rpm-build rpm-devel
 

Proliant Support Package is not supported on CentOS so you have to let the installer think it is a RedHat system. If you have a RHEL system skip the next steps.

# cp /etc/redhat-release /etc/redhat-release.backup

# echo "Red Hat Enterprise Linux Server release 5.4 (Tikanga)">/etc/redhat-release

Untar the Proliant Support Package

 # tar xzvf psp-8.25.rhel5.i686.en.tar.gz

Install the Proliant Support Package

# cd compaq/csp/linux/

# ./install825.sh

alot of text appears and some questions, answer them.

Install HP System Health Application and Insight Management Agents, and for some stupid reason it is in conflict with some other packages just installed. I solved it in a dirty way:

# rpm -i --force  --replacefiles --nodeps hpasm-8.0.0-173.rhel5.i386.rpm

Configure by running:

# hpa/etc/init.d/hpasm configure

and answer the questions.

Do not forget to restore /etc/redhat-release

# cp /etc/redhat-release.backup /etc/redhat-release

I did modify my /etc/snmp/snmp.conf to:

dlmod cmaX /usr/lib/libcmaX.so
rocommunity public
trapsink 10.1.1.20
syscontact peter@it-slav.net
syslocation PDC, Peters DataCenter

To test that you have installation and configuration work, run a snmpwalk from your Nagios or op5 Monitor host:

# snmpwalk -c public -v1  <ip-adress of your proliant box> 1.3.6.1.4.1.232

SNMPv2-SMI::enterprises.232.1.1.1.0 = INTEGER: 1
SNMPv2-SMI::enterprises.232.1.1.2.0 = INTEGER: 23
SNMPv2-SMI::enterprises.232.1.1.3.0 = INTEGER: 2
SNMPv2-SMI::enterprises.232.1.2.1.4.1.0 = INTEGER: 30
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.1.1 = INTEGER: 1
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.2.1 = STRING: "Compaq Standard Equipment Agent for Linux"
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.3.1 = ""
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.4.1 = Hex-STRING: 00 00 00 00 00 00 00
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.5.1 = STRING: "To gather Standard Equipment data for Linux."

...
 

Install check_hpasm on the Nagios or op5 Monitor host

The check_hpasm can be downloaded from Console Labs.

Unpack the tarball

# tar xzvf check_hpasm-4.1.2.tar.gz

Configure and compile

# ./configure --prefix=/opt/plugins/custom/hp-insight --with-nagios-user=monitor --with-nagios-group=users --enable-perfdata

...

# make

...

# make install

Test

# /opt/plugins/custom/hp-insight/libexec/check_hpasm -H  -C public

OK - System: 'proliant dl360 g3', S/N: '7J31LMW6N01D', ROM: 'P31 01/28/2004', hardware working fine, da: 1 logical drives, 1 physical drives | fan_1=50% fan_2=50% temp_1_cpu=16;50;50 temp_2_cpu=15;65;65 temp_3_ioBoard=21;56;56 temp_4_cpu=20;65;65

Congratulations your plugin and hw monitoring works!

Configure Nagios or op5 Monitor
 

checkcommands.cfg

# command 'check_hpasm'
define command{
    command_name                   check_hpasm
    command_line                   $USER1$/custom/libexec/check_hpasm -H $HOSTADDRESS$ -C $ARG1$
    }

services.cfg

# service 'Insight Manager'
define service{
    use                            default-service
    host_name                      humpa
    service_description            Insight Manager
    check_command                  check_hpasm!public
    contact_groups                 call_it-slav,it-slav_jabber,it-slav_mail
    }

Screenshoot, using ninja

 Useful links

• op5, a systems and network management company
• op5 Monitor, an enterpise monitor system based on Nagios
• Ninja, Nagios is now just awesome
• Nagios, enterprise monitoring based on opensource
• Hp support & drivers, a place to start looking for the HP software used in this article

Background

At op5 we often run into BizTalk at customer sites as a business critical application. We have done an investigation and come up with a plugin  to monitor BizTalk.

To fully understand this document BizTalk knowledge is necessary.

Solution

op5 has developed a plugin that can monitor the following at a BizTalk server:

• Recieve locations – Verify that they are enabled.
• Send Ports – Verify that they are in Started state
• Group Overview – Catch messages in the message queue that are in state Suspended. The different type of messages are:
  • Dehydrated
  • Suspended
  • Queue

The plugin license is GPLv3 so it is free for any one to use and enhance.

Links

• check_biztalk the plugin that monitors BizTalk
• op5 Monitor Monitor solution based on Nagios
• Nagios free enterprise monitor software

I was reading the Swedish computer magazine, Tech World. There was an article about an appliance solution that could give a report about unused switch ports. The box PortIQ from Infoblox costs approximatly 10.000 Euro.

op5 has developed this feature at a customer site and is included with op5 Monitor, op5 has published the code as opensource.

This guide will describe howto get it running in op5 Monitor. op5 Monitor is vanilla Nagios with some addons so it should be no problem to follow this guide in a Nagios environment

Installation

1. Update your op5 Monitor or download the plugin and report script.

2. Create metadata for check_portstatus as $USER1$/check_portstatus -H $HOSTADDRESS$

in checkcommands.cfg:

# command 'check_portstatus'
define command{
    command_name                   check_portstatus
    command_line                   $USER1$/check_portstatus -H $HOSTADDRESS$
    }
 

3. mkdir /var/spool/check_portstatus

4. chown monitor:apache /var/spool/check_portstatus

5. cp /opt/monitor/op5/portstatus/portstatus.php /var/www/html/portstatus.php

6. Define your service i.e. services.cfg:

# service 'Portstatus'
define service{
    use                            default-service
    host_name                      gw
    service_description            Portstatus
    check_command                  check_portstatus
    }
 

The result

The screenshot below shows a Cisco switch with alot of unused ports.

Conclusions

This article shows how easy it is to modify Nagios and other opensource tools to give you what you want. Do not hesitate to contact me or op5 to help you enhance you Nagios or op5 installation.

Links

• op5
• op5 Monitor
• Nagios
• PortIQ appliance box from Infoblox
• Tech world

This article will describe how easy it is to integrate other tools to Nagios or op5 Monitor. I will use an example with a webshop where a business view of how the webshop is doing is implemented by using a GPL’d rule engine, NodeBrain. I have in an earlier article described the ruleset for this implementation but now I will show how the integration can be done.

Scenario

The scenario is a webshop with:

• 5 frontend webservers
• 2 application servers
• 3 databasservers

Management want to monitor how the webshop is doing. They do not want to know if a redundant part is down instead management want to have the overview of the webshop status.

A management consultant is hired and do an investigation and after a ridiculous amount of money the following rules are defined:

• Webserver rules
  • If 3 or more webserver works the webservice is OK
  • If 2 webservers works the webservice is WARNING
  • If 1 webserver or less is working the webservice is CRITICAL
• Applicationserver rules
  • If 1 or 2 application servers works the application layer is OK
  • If zero application servers works the application layer is CRITICAL
• Database server rules
  • If 2 or more database server works the database layer is OK
  • if 1 database server works the database layer is WARNING
  • If no database servers works the database layer is CRITICAL
•  The webserver layer, application layer and database layer should be viewed seperatly
• The total webshop status has the highest status value of webserver layer, application layer and database layer

I use Nagvis to illustrate the releationship between the layers.

Case 1

The picture shows when everything is fine:

Case 2

The rules in place:

• 2 webservers CRITICAL and the webserverlayer is OK
• 1 appserver CRITICAL and the application layer is OK
• 2 databases are down and the database layer is WARNING
• The total webshop status is WARNING because it has the highest status of the different layers

Case 3

Now it has been even worse:

• 4 webservers CRITICAL and the webserverlayer is CRITICAL
• 1 appserver CRITICAL and the application layer is OK
• 2 databases are down and the database layer is WARNING
• The total webshop status is CRITICAL because it has the highest status of the different layers

Conclusions

This article shows the power of Open Source and what is possible todo when integration different projects with each other. A solution like this with one of the Big Four (IBM, BMC, CA, HP) would have cost alot when it comes to license and highly specialised consultants had to be hirered.

Links

• Op5 A company that package and support enterprise class systems- and networkmanagement products
• NodeBrain, A powerful GPL’d rule engine
• Nagios, An enterprise class Monitoring sofware
• Nagvis, Nagios Visualization addon

Implementation

Hosts and services

The hosts and services are created:

• Webserver layer: 5 hosts each with 1 service
• Application layer: 2 hosts each with 1 service
• Database layer: 3 hosts each with 1 service
• Webshop layer: 1 host called webshopcontainer and 4 services: webserversstatus, appsserverstatus, dbserversstatus and webshop status. The services represent each layer in the model and webshop is the total status of the webshop.

To make it easy to control the status of all these devices I will use passive checks. So if I want to change the status of a service I just use the GUI and send in a passive check result. In real life active checks would have been used to monitor the different services.

The result showing Service Detail using op5 developed Nagios GUI Ninja:

The statechanges are sent to NodeBrain via an eventhandler that sends the NodeBrain commands via a namedpipe:

#!/bin/sh
#
# Event handler script for sending nagios data to nodebrain
#
# This script has these arguments $SERVICESTATEID$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $SERVICEDESC$ $HOSTNAME$
NODEBRAINPIPE=/opt/plugins/custom/nodebrainpipe
DATE=`date`
echo "$DATE SERVICESTATEID=$1 SERVICESTATETYPE=$2 SERVICEATTEMPT=$3 SERVICEDESC=$4 HOSTNAME=$5" >> /tmp/eventhandler_out

exit 0

The Nodebrainrules, that has all the logic and opens the namedpipe for commands:

#!/usr/local/bin/nb -d
#
-rm webshop.log
set log="webshop.log",out=".";
declare indata identity owner;
define webshop node pipe.server("indata@nodebrainpipe");

#Total rules
assert webshopstatus=0;
#If all serverstatus ok, the whole webshop is ok
define webshopok on(webserversstatus=0 and appserversstatus=0 and dbserversstatus=0) webshopstatus=0;
define webshopoksend on(webshopstatus=0):-./send_to_monitor.sh webshopstatus 0 "OK: Webshop is fine";
#If any serverstatus is critical the whole webshop is critical
define webshopscritical on(webserversstatus=2 or appserversstatus=2 or dbserversstatus=2) webshopstatus=2;
define webshopscriticalsend on(webshopstatus=2):-./send_to_monitor.sh webshopstatus 2 "CRITICAL: Webshop has serious problems";
#If not any serverstatuscritical and in warning, the whole shop is warning.
define webshopwarning on((!webserversstatus=2 and !appserversstatus=2 and !dbserversstatus=2) and (webserversstatus=1 or dbserversstatus=1)) webshopstatus=1;
define webshopwarningsend on(webshopstatus=1):-./send_to_monitor.sh webshopstatus 1 "WARNING: Webshop has some problems";
 

The NodeBrain rules runs this script when fired:

#!/bin/sh

HOSTNAME=webshopcontainer
SERVICEDESC=$1
STATUS=$2
MESSAGE=$3


now=`date +%s`
commandfile='/opt/monitor/var/rw/nagios.cmd'
/usr/bin/printf "[%lu] PROCESS_SERVICE_CHECK_RESULT;$HOSTNAME;$SERVICEDESC;$STATUS;$MESSAGE\n" $now > $commandfile

The Nagios or op5 Monitor hosts.cfg

###############################################################################
#  Generated by op5 Monitor webconfiguration exporter
#
#  Exported 2009-10-22 19:33 by monitor
#

# host 'webshopcontainer'
define host{
    use                            Dummy-template
    host_name                      webshopcontainer
    alias                          webshopcontainer
    address                        127.0.0.1
    contact_groups                 support-group
    }
 

The Nagios or op5 Monitor services.cfg

###############################################################################
#  Generated by op5 Monitor webconfiguration exporter
#
#  Exported 2009-10-22 19:33 by monitor
#

# service 'webshopstatus'
define service{
    use                            Dummy-service-template
    host_name                      webshopcontainer
    service_description            webshopstatus
    servicegroups                  webshop
    flap_detection_enabled         0
    }

Next week, op5 will release the new Nagios GUI called Ninja. The license is GPL and several people has reported the GUI work perfectly well on vanilla Nagios. Ninja uses a database backend. Merlin, another op5 developed software feeds the Nagios information into the database.

Links:

• op5 -OpenSource based management software company
• Ninja- Nagios Is Now Just Awesome
• Merlin – Module for Effortless Redundancy and Loadbalancing In Nagios
• Try – Ninja GUI

Some screenshoots below:

 Geomap, based on google maps

Tactical overview, with widgets that could be dragged and dropped

Host details

Nagvis integration

http://www.thegeekstuff.com/2009/09/top-5-best-network-monitoring-tools/

By installing wp-cache the testcase went down to 0.4 seconds.

 The drawback is that my counter widget is getting updated every hour instead of every reload of the webpage.

Take a peak at some nice graphs showing that I stopped the increasing loading time and got a speedup from 7.5 seconds to 0.4 seconds:

http://wiki.nagios.org/index.php/SkypeSmsNotification

My recommendation is to use an GSM modem to send SMS, because if the network is down no SMS can be sent with this solution and SMS is normally used as an alternative alarm route.

Another aspect is that Skype is using propertiary protocols and that should be avoided.

I use MythTV quite frequently and noticed that it is instable when using sasc-ng as a decoder to decrypt encrypted DVB-T channels. So approximatly every third day the MythTVbackend server stops and need to be started again. I have wriiten an earlier article about howto monitor MythTV with Nagios or op5 Monitor so I get noticed that it has stopped. But I need to manually start it again. This article describe howto make Nagios or op5 Monitor to start a stopped MythTVbackend. It can be used for starting almost any service.

I have used the examples provided by Ethan at Nagios official documentation describing eventhandlers.

Normally it is not recommended to let a tool like Nagios or op5 Monitor start a service that has stopped, because it is probably a reason why the service has stopped and the correct procedure is to fix the root cause of the problem, not the symptom.

The MythTV backend runs on one machine called lala (after a character in Teletubbies) which is not the same as the Nagios or op5 Monitor server. I use nrpe to run the start script i.e.

 /etc/init.d/mythtv-backend start

There is several options here but I already setup the nrpe agent and it is simple to make Nagios or op5 Monitor to use nrpe to run a script.

Implementation

I used the script I found at Nagios documentation about eventhandlers as a base and modiied it slightly.

At my op5 Monitor machine

/opt/plugins/custom/restart-mythtv-lala.sh

#!/bin/sh
#
# Event handler script for restarting the mythTVbackend server on lala
#
# Note: This script will only restart the mythtvbackend if the service is
#       retried 2 times (in a "soft" state) or if the service somehow
#       manages to fall into a "hard" error state.
#

# What state is the mythbackend service in?
case "$1" in
OK)
	# The service just came back up, so don't do anything...
	;;
WARNING)
	# We don't really care about warning states, since the service is probably still running...
	;;
UNKNOWN)
	# We don't know what might be causing an unknown error, so don't do anything...
	;;
CRITICAL)
	# Aha!  The HTTP service appears to have a problem - perhaps we should restart the server...

	# Is this a "soft" or a "hard" state?
	case "$2" in

	# We're in a "soft" state, meaning that Nagios is in the middle of retrying the
	# check before it turns into a "hard" state and contacts get notified...
	SOFT)

		# What check attempt are we on?  We don't want to restart the web server on the first
		# check, because it may just be a fluke!
		case "$3" in

		# Wait until the check has been tried 3 times before restarting the web server.
		# If the check fails on the 4th time (after we restart the web server), the state
		# type will turn to "hard" and contacts will be notified of the problem.
		# Hopefully this will restart the web server successfully, so the 4th check will
		# result in a "soft" recovery.  If that happens no one gets notified because we
		# fixed the problem!
		2)
			echo "`date` Restarting mythtv service (2rd soft critical state)..." >> /tmp/mythtvstart
			# Call the init script to restart the mythbackend server
			#/etc/rc.d/init.d/httpd restart
			#date >> /tmp/mythtvstart
			/opt/plugins/check_nrpe -H lala -c start_mythtvbackend
			;;
			esac
		;;

	# The mythtvbackend service somehow managed to turn into a hard error without getting fixed.
	# It should have been restarted by the code above, but for some reason it didn't.
	# Let's give it one last try, shall we?
	# Note: Contacts have already been notified of a problem with the service at this
	# point (unless you disabled notifications for this service)
	HARD)
		echo "`date` Restarting mythtv service (hard state)..." >> /tmp/mythtvstart
		# Call the init script to restart the HTTPD server
		#/etc/rc.d/init.d/httpd restart
		#date >> /tmp/mythtvstart
		/opt/plugins/check_nrpe -H lala -c start_mythtvbackend
		;;
	esac
	;;
esac
exit 0

/opt/monitor/misccomands.cfg

# command 'restart-mythtv-lala'
define command{
    command_name                   restart-mythtv-lala
    command_line                   /opt/plugins/custom/start-mythtv-lala.sh $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$
    }

/opt/monitor/etc/services.cfg

# service 'Mythbackend'
define service{
    use                            default-service
    host_name                      lala
    service_description            Mythbackend
    check_command                  check_tcp!6543
    servicegroups                  MythTV,it-slav
    event_handler                  restart-mythtv-lala!$SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$
    contact_groups                 it-slav_sms,it-slav_jabber,it_slav_mail
    }

At my mythbackend machine lala

/etc/nrpe.d/mycommands.cfg
command[start_mythtvbackend]=/usr/bin/sudo /etc/init.d/mythtv-backend start

/etc/sudoers
nobody ALL= (root) NOPASSWD:/etc/init.d/mythtv-backend start

Test

I stopped the mythtvbackend by running:

peter@lala:/etc/nrpe.d$ date
Mon Jun 15 20:40:55 CEST 2009
peter@lala:/etc/nrpe.d$ sudo /etc/init.d/mythtv-backend stop
 * Stopping MythTV server: mythbackend

And run

[root@op5 ~]# tail -f /tmp/mythtvstart
Mon Jun 15 20:47:09 CEST 2009 Restarting mythtv service (2rd soft critical state)...

YES it works!

Links:

• op5 Monitor a Nagios based supported enterprise Monitoring software.
• MythTV a free OpenSource Digital Video Recorder
• Nagios Open Source Monitoring

http://www.op5.org/community/projects/op5-plugins

The plugins is also located in a Git repository and can be checked out from:

http://git.op5.org/git/

• Tobias Oitiker gave a very good presentation about the linux kernel caching and described how it affected RRDtools.
• Markus Almroth, describes Sysnagios which did the opposite of most monitoring projects. Sysnagios decentralized the configuration of systems and network monitoring in a controlled way. I will come back to this topic in my blog.
• Per Åsberg talked about Ninja, a new GUI for Nagios, done by op5.
• Andriy Skuratov, talked about and demonstrated a new version of Nagvis with integrated Google Maps and other good stuff.
• Ethan Galstad the founder of Nagios announced Ninja as the new GUI for Nagios.

During the Nordic meet on Nagios I meet alot of interesting people, learned alot and had good food and drinks.

I hope to see you next year at Nordic Meet on Nagios 2010!!!

Ethan Galstad, the founder of Nagios has done a great job to open up the project and listening to input from the community.

Go to http://ideas.nagios.org/ and vote or add the functionality you want in Nagios so others can vote for it.

I have added the idea that Nagios should have a programable rule engine. When I worked as a Tivoli consultant it was essential to collect information from different sources and write som rules and present the result. All the enterprise monitoring solutions has a rule engine, i.e PEM (Patrol Enterprise Manager), T/EC (Tivoli Enterprise Console) and so on.

Read more at http://www.op5.org/community/projects/merlin

I have just finnished reading the excellent book "Learning Nagios 3.0" by Wojciech Kocjan.

The book does not require any prior knowledge about Nagios but having basic knowledge in programing, Unix or Linux is good. I as a rather experienced Nagios admin did get some valuable hints by reading it. I really liked the chapter about differences between Nagios 2 and 3.

The books target audience is sysadmins with Unix or LInux background that want to Monitor their environment by using Nagios. For a beginner going to the Nagios homepage the information is overvelming and it has to know where to start, a book like this is handy. The book often refers to specific topics at Nagios documentation for more info and that is good if the reader wants to dig deeper into that topic. Nagios homepage is an excellent reference but not a place to start when you want to use Nagios.

The examples are good and there is alot of illustrations, sometimes the illustrations is more complicated then the topic itself. The writer shows deep knowledge about the subject and has done a good job getting it technically correct, sometimeas a little bit academic i.e. it is more important to get everything correct then explaing something in a clear way.

I would recommend anybody interested in using Nagios to buy and read this book, especialy if you do not have any prior knowledge to Nagios.

Links:

• To buy Learning Nagios 3.0
• Example from the book, Chapter 7 passive checks and nsca

Valued Nagios users and Community members -

As many of you are already aware, there has been a great deal of
activity surrounding the recent announcement of a Nagios fork.  There's
a lot at stake with the outcome of this fork and its impact on the
Nagios community.  Rather than spend my time committing bug fixes to
CVS, I felt it was a much wiser choice for me to share my thoughts on
this issue with all of you.

You can read my thoughts on the fork and its impact on the future of
Nagios at http://community.nagios.org

A direct link to my main post:

http://community.nagios.org/2009/05/11/nagios-a-fork-in-the-road/

The short version is this - Nagios is alive, the community is vibrant,
and the future looks bright.

- Ethan Galstad, Creator/Father/Defender of Nagios

Andreas Ericson one of the members of the Nagios steering board has written this to his blog:

Some of you might know that a fork of Nagios has appeared recently. If you don’t, go read about it in the nagios-devel mailing list archives. They’re available on sourceforge somewhere, but I can’t be bothered to look for them right now.

Working for a company that makes a living out of supporting and writing addons for Nagios, I must say I’m a bit sad. Being an enthusiastic and optimistic guy, I must say I’m thrilled.

A couple of facts before we set off:

• The fork was instigated largely by german members of the community. It appears to have been spearheaded by a german company (though I don’t know this for sure) that makes its living selling customized Nagios solutions and/or support. I don’t know this for sure, but it sure looks as if that’s what’s happened.
• The german company have unlawfully used the Nagios trademark after being asked not to do so. It has also registered Nagios as a trademark in Germany, to which is a huge slap in the face of an opensource project. They are naturally not on the best of terms with Nagios’ founding father, Ethan, at the moment.
• Ethan has been absent working with the aforementioned lawsuit (or whatever it is a trademark violation results in when friendly talk is no longer enough), and also trying to put together a new webbased user interface for Nagios.
• Patches from all levels of the community have been erratically ignored during Ethan’s absence. Some were picked up, but as many or more slipped between the cracks.
• Ethan has always been the single person with commit access to the Nagios CVS (yuk) repository.
• The fork uses git to track their patches.

The community developers have voiced a complaint that they cite as the primary reason for the fork:

Nagios is not being developed fast and openly enough.

I agree with this, and I’m currently discussing with Ethan about expanding the developer-base. Unfortunately, the scarce resource “trust” is even scarcer for those developers who joined the fork, which leaves the available candidates rather few. Happily, I count myself among them, and apparently so does Ethan. He emailed me away from public channels asking if I’d be willing to become a core developer, and op5 has graciously given a tentative promise to devote one to two days per week to Nagios development / patch management. Nothing’s settled yet, but development has to continue even if the core maintainer takes a leave of absence, so one way or another, we’ll make sure this happens.

In a perfect world (ie, one where I get to decide everything ), here’s what will happen:

• Nagios incorporates the good changes that the fork produces.
• The benevolent but previously frustrated developers from the fork hop back to working on Nagios when they see it’s once again moving forward. They could actually do that by keeping on working on their fork, although that would set them apart from the Nagios community a bit rather than make them members of it.
• Nagios development picks up its pace and a new GUI is added to it which fulfills everyones wildest dreams.
• Nagios development moves to using git instead of CVS. Since git actually invites people to fork the code but makes it incredibly simple to merge those changes back to the pre-fork project again, there could be any number of forks and Nagios would be the grand total of the best of all of them. Who would win on that? Well, the Nagios users for a start, and Nagios itself, and Ethan, and every company making a living off of Nagios one way or another. So that’d be a win-win-win-win-win situation? I like it.

For those who wonder where I’m standing in all this, I’ll be working with Ethan to make the community developers happy while at the same time trying to prevent the community users from living through the confusion that a long-lived fork means. In the end, I hope Nagios becomes a better product with a stronger and better community backing it, which seems rather inevitable now that more people than ever are working frantically at making it so. Hopefully it results in a happy community where The Right People(tm) are part of a Nagios steering committee or some such.

Time will tell. It always does