I have registred me to attend the open Splunk event in Stockholm. Within 24 hours I got an email telling that they do not want competitors at the event and would like to discuss the purpose of me attending.

I called the Splunk representative and told him that op5 thinks Splunk is a competitor in some cases and in some cases we have products that run in parallel at customer sites. We want to integrate op5 Monitor with Splunk because we have got the question a couple of times from customers and potential customers. My intention of attending the meeting is to get a better understanding of Splunk so op5 can integrate our products, so our customers get the products that suits them best. If Splunk wants to be part of the integration, the better.

The Splunk representative still do not want me to attend. At Splunk they will discuss this matter internally but so far, this is their view.

I would really like to make a parallel with another op5 competitor, Würth, at their event they invited op5 as key speaker and I really recommend Splunk to reconsider their decision.

With my new and fancy HTC Desire I wanted to look at my Nagios or op5 Monitor status. I have found two apps Nagroid and NagMonDroid.

NagMonDroid did I not get working, probably because op5 Monitor only uses https as protocol.

Nagroid works like a charm:

The configuration is rather straightforward:

For the first time I felt that this is more than a phone, for the first time calender integration works, for the first time I can use the builtin GPS, for the first time accessing the web with a phone works, for the first time downloaded software really works.

My favorite apps so far is:

• Nagroid, to be able to view my op5 Monitor status
• FONMaps, find hotspots for LaFoneras
• Car Cast, listen and download podcasts
• MapDroid, to use preloaded OpenStreetmaps and GPS without using any bandwidth. Perfect when abroad because of the crazy price of data roaming outside Sweden.
• HTC Facebook, read and post on facebook
• FON Access, automatically connect to FON hotspots when traveling.
• GPS Logger, logg tracks in gpx format that almost any GPS software understand
• Sipdroid, to connect to my Asterisk PBX using 3G or WiFi

Of  cource the device is not perfect, I miss:

• The phone must be ‘rooted’ to be real useful, why? Open the phone so the community and others can develop apps that are real useful. Vendor lock-in always sucks.
• IPSec VPN so I can connect to my IPSec based OpenBSD firewall. The IPSec implementation in the phone sucks.
• OpenVPN, there exists OpenVPN apps but to use the the phone must be ‘rooted’
• Bluetooth modem, it is not possible to use the phone as a modem using bluetooth
• Screenshots, the phone must be rooted or using the SDK to take screenshoots. Why?

I really hope that Google and/or HTC understand and use the power of the community to make the Android even more succesfull by open it even more.

Background

It is always a good idea to monitor the server hardware, in many cases the root cause of the probblem is hardware related like: a fan stops and the temperature gets to high, dust in the machine makes it to hot, disks that fails, memory corruption and so on.  This article will describe howto enable hardware monitoring on a HP Proliant running CentOS Linux and then howto collect the data with Nagios or op5 Monitor. The procedure is the same with RedHat Enterprise Linux and similiar with Suse Enterprise server.

The HP manuals and information is bloated with irrelevant information and I had to struggle several hours, ask collegues to get it running. I hope this blog article will help others to get monitoring of HP Proliant using HP Insight Manager easier to setup.

Installing the software on the target system

You need two packages from HP and can be downloaded from hp.com under "Support & Drivers". Search your hardware plattform and correct operating system:

• ProLiant Support Pack for Red Hat Enterprise Linux 5 (i686)  The latest Proliant Support Package 2010-03-02 name is psp-8.25.rhel5.i686.en.tar.gz
• HP System Health Application and Insight Management Agents for Red Hat Enterprise Linux 5 (x86). The latest name is hpasm-8.0.0-173.rhel5.i386.rpm

Install kernel source code and rpm tools:

# yum install kernel-devel rpm-build rpm-devel
 

Proliant Support Package is not supported on CentOS so you have to let the installer think it is a RedHat system. If you have a RHEL system skip the next steps.

# cp /etc/redhat-release /etc/redhat-release.backup

# echo "Red Hat Enterprise Linux Server release 5.4 (Tikanga)">/etc/redhat-release

Untar the Proliant Support Package

 # tar xzvf psp-8.25.rhel5.i686.en.tar.gz

Install the Proliant Support Package

# cd compaq/csp/linux/

# ./install825.sh

alot of text appears and some questions, answer them.

Install HP System Health Application and Insight Management Agents, and for some stupid reason it is in conflict with some other packages just installed. I solved it in a dirty way:

# rpm -i --force  --replacefiles --nodeps hpasm-8.0.0-173.rhel5.i386.rpm

Configure by running:

# hpa/etc/init.d/hpasm configure

and answer the questions.

Do not forget to restore /etc/redhat-release

# cp /etc/redhat-release.backup /etc/redhat-release

I did modify my /etc/snmp/snmp.conf to:

dlmod cmaX /usr/lib/libcmaX.so
rocommunity public
trapsink 10.1.1.20
syscontact peter@it-slav.net
syslocation PDC, Peters DataCenter

To test that you have installation and configuration work, run a snmpwalk from your Nagios or op5 Monitor host:

# snmpwalk -c public -v1  <ip-adress of your proliant box> 1.3.6.1.4.1.232

SNMPv2-SMI::enterprises.232.1.1.1.0 = INTEGER: 1
SNMPv2-SMI::enterprises.232.1.1.2.0 = INTEGER: 23
SNMPv2-SMI::enterprises.232.1.1.3.0 = INTEGER: 2
SNMPv2-SMI::enterprises.232.1.2.1.4.1.0 = INTEGER: 30
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.1.1 = INTEGER: 1
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.2.1 = STRING: "Compaq Standard Equipment Agent for Linux"
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.3.1 = ""
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.4.1 = Hex-STRING: 00 00 00 00 00 00 00
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.5.1 = STRING: "To gather Standard Equipment data for Linux."

...
 

Install check_hpasm on the Nagios or op5 Monitor host

The check_hpasm can be downloaded from Console Labs.

Unpack the tarball

# tar xzvf check_hpasm-4.1.2.tar.gz

Configure and compile

# ./configure --prefix=/opt/plugins/custom/hp-insight --with-nagios-user=monitor --with-nagios-group=users --enable-perfdata

...

# make

...

# make install

Test

# /opt/plugins/custom/hp-insight/libexec/check_hpasm -H  -C public

OK - System: 'proliant dl360 g3', S/N: '7J31LMW6N01D', ROM: 'P31 01/28/2004', hardware working fine, da: 1 logical drives, 1 physical drives | fan_1=50% fan_2=50% temp_1_cpu=16;50;50 temp_2_cpu=15;65;65 temp_3_ioBoard=21;56;56 temp_4_cpu=20;65;65

Congratulations your plugin and hw monitoring works!

Configure Nagios or op5 Monitor
 

checkcommands.cfg

# command 'check_hpasm'
define command{
    command_name                   check_hpasm
    command_line                   $USER1$/custom/libexec/check_hpasm -H $HOSTADDRESS$ -C $ARG1$
    }

services.cfg

# service 'Insight Manager'
define service{
    use                            default-service
    host_name                      humpa
    service_description            Insight Manager
    check_command                  check_hpasm!public
    contact_groups                 call_it-slav,it-slav_jabber,it-slav_mail
    }

Screenshoot, using ninja

 Useful links

• op5, a systems and network management company
• op5 Monitor, an enterpise monitor system based on Nagios
• Ninja, Nagios is now just awesome
• Nagios, enterprise monitoring based on opensource
• Hp support & drivers, a place to start looking for the HP software used in this article

• op5 Monitor Beta, virtual image in ovf format
• Try op5 Monitor Beta

The op5 Monitor Beta use Ninja as GUI and Merlin as a databasebackend.

–cut–

• Implementation of the missing Nagios cgi’s. (showlog, history,trends, notifications etc)
• Include Avail/SLA reports in Ninja  (Avail reports are an improved version of the Nagios avail reports and SLA is a new report type)
• Lots of bugfixes

–cut–

Links

• Ninja, Nagios Is Now Just Awesome
• Merlin, Module for Endless Redundacy and Loadbalancing In Nagios
• op5, a company with enterprise network- and systems management based on OpenSource.

Background

At op5 we often run into BizTalk at customer sites as a business critical application. We have done an investigation and come up with a plugin  to monitor BizTalk.

To fully understand this document BizTalk knowledge is necessary.

Solution

op5 has developed a plugin that can monitor the following at a BizTalk server:

• Recieve locations – Verify that they are enabled.
• Send Ports – Verify that they are in Started state
• Group Overview – Catch messages in the message queue that are in state Suspended. The different type of messages are:
  • Dehydrated
  • Suspended
  • Queue

The plugin license is GPLv3 so it is free for any one to use and enhance.

Links

• check_biztalk the plugin that monitors BizTalk
• op5 Monitor Monitor solution based on Nagios
• Nagios free enterprise monitor software

I was reading the Swedish computer magazine, Tech World. There was an article about an appliance solution that could give a report about unused switch ports. The box PortIQ from Infoblox costs approximatly 10.000 Euro.

op5 has developed this feature at a customer site and is included with op5 Monitor, op5 has published the code as opensource.

This guide will describe howto get it running in op5 Monitor. op5 Monitor is vanilla Nagios with some addons so it should be no problem to follow this guide in a Nagios environment

Installation

1. Update your op5 Monitor or download the plugin and report script.

2. Create metadata for check_portstatus as $USER1$/check_portstatus -H $HOSTADDRESS$

in checkcommands.cfg:

# command 'check_portstatus'
define command{
    command_name                   check_portstatus
    command_line                   $USER1$/check_portstatus -H $HOSTADDRESS$
    }
 

3. mkdir /var/spool/check_portstatus

4. chown monitor:apache /var/spool/check_portstatus

5. cp /opt/monitor/op5/portstatus/portstatus.php /var/www/html/portstatus.php

6. Define your service i.e. services.cfg:

# service 'Portstatus'
define service{
    use                            default-service
    host_name                      gw
    service_description            Portstatus
    check_command                  check_portstatus
    }
 

The result

The screenshot below shows a Cisco switch with alot of unused ports.

Conclusions

This article shows how easy it is to modify Nagios and other opensource tools to give you what you want. Do not hesitate to contact me or op5 to help you enhance you Nagios or op5 installation.

Links

• op5
• op5 Monitor
• Nagios
• PortIQ appliance box from Infoblox
• Tech world

This article will describe how easy it is to integrate other tools to Nagios or op5 Monitor. I will use an example with a webshop where a business view of how the webshop is doing is implemented by using a GPL’d rule engine, NodeBrain. I have in an earlier article described the ruleset for this implementation but now I will show how the integration can be done.

Scenario

The scenario is a webshop with:

• 5 frontend webservers
• 2 application servers
• 3 databasservers

Management want to monitor how the webshop is doing. They do not want to know if a redundant part is down instead management want to have the overview of the webshop status.

A management consultant is hired and do an investigation and after a ridiculous amount of money the following rules are defined:

• Webserver rules
  • If 3 or more webserver works the webservice is OK
  • If 2 webservers works the webservice is WARNING
  • If 1 webserver or less is working the webservice is CRITICAL
• Applicationserver rules
  • If 1 or 2 application servers works the application layer is OK
  • If zero application servers works the application layer is CRITICAL
• Database server rules
  • If 2 or more database server works the database layer is OK
  • if 1 database server works the database layer is WARNING
  • If no database servers works the database layer is CRITICAL
•  The webserver layer, application layer and database layer should be viewed seperatly
• The total webshop status has the highest status value of webserver layer, application layer and database layer

I use Nagvis to illustrate the releationship between the layers.

Case 1

The picture shows when everything is fine:

Case 2

The rules in place:

• 2 webservers CRITICAL and the webserverlayer is OK
• 1 appserver CRITICAL and the application layer is OK
• 2 databases are down and the database layer is WARNING
• The total webshop status is WARNING because it has the highest status of the different layers

Case 3

Now it has been even worse:

• 4 webservers CRITICAL and the webserverlayer is CRITICAL
• 1 appserver CRITICAL and the application layer is OK
• 2 databases are down and the database layer is WARNING
• The total webshop status is CRITICAL because it has the highest status of the different layers

Conclusions

This article shows the power of Open Source and what is possible todo when integration different projects with each other. A solution like this with one of the Big Four (IBM, BMC, CA, HP) would have cost alot when it comes to license and highly specialised consultants had to be hirered.

Links

• Op5 A company that package and support enterprise class systems- and networkmanagement products
• NodeBrain, A powerful GPL’d rule engine
• Nagios, An enterprise class Monitoring sofware
• Nagvis, Nagios Visualization addon

Implementation

Hosts and services

The hosts and services are created:

• Webserver layer: 5 hosts each with 1 service
• Application layer: 2 hosts each with 1 service
• Database layer: 3 hosts each with 1 service
• Webshop layer: 1 host called webshopcontainer and 4 services: webserversstatus, appsserverstatus, dbserversstatus and webshop status. The services represent each layer in the model and webshop is the total status of the webshop.

To make it easy to control the status of all these devices I will use passive checks. So if I want to change the status of a service I just use the GUI and send in a passive check result. In real life active checks would have been used to monitor the different services.

The result showing Service Detail using op5 developed Nagios GUI Ninja:

The statechanges are sent to NodeBrain via an eventhandler that sends the NodeBrain commands via a namedpipe:

#!/bin/sh
#
# Event handler script for sending nagios data to nodebrain
#
# This script has these arguments $SERVICESTATEID$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $SERVICEDESC$ $HOSTNAME$
NODEBRAINPIPE=/opt/plugins/custom/nodebrainpipe
DATE=`date`
echo "$DATE SERVICESTATEID=$1 SERVICESTATETYPE=$2 SERVICEATTEMPT=$3 SERVICEDESC=$4 HOSTNAME=$5" >> /tmp/eventhandler_out

exit 0

The Nodebrainrules, that has all the logic and opens the namedpipe for commands:

#!/usr/local/bin/nb -d
#
-rm webshop.log
set log="webshop.log",out=".";
declare indata identity owner;
define webshop node pipe.server("indata@nodebrainpipe");

#Total rules
assert webshopstatus=0;
#If all serverstatus ok, the whole webshop is ok
define webshopok on(webserversstatus=0 and appserversstatus=0 and dbserversstatus=0) webshopstatus=0;
define webshopoksend on(webshopstatus=0):-./send_to_monitor.sh webshopstatus 0 "OK: Webshop is fine";
#If any serverstatus is critical the whole webshop is critical
define webshopscritical on(webserversstatus=2 or appserversstatus=2 or dbserversstatus=2) webshopstatus=2;
define webshopscriticalsend on(webshopstatus=2):-./send_to_monitor.sh webshopstatus 2 "CRITICAL: Webshop has serious problems";
#If not any serverstatuscritical and in warning, the whole shop is warning.
define webshopwarning on((!webserversstatus=2 and !appserversstatus=2 and !dbserversstatus=2) and (webserversstatus=1 or dbserversstatus=1)) webshopstatus=1;
define webshopwarningsend on(webshopstatus=1):-./send_to_monitor.sh webshopstatus 1 "WARNING: Webshop has some problems";
 

The NodeBrain rules runs this script when fired:

#!/bin/sh

HOSTNAME=webshopcontainer
SERVICEDESC=$1
STATUS=$2
MESSAGE=$3


now=`date +%s`
commandfile='/opt/monitor/var/rw/nagios.cmd'
/usr/bin/printf "[%lu] PROCESS_SERVICE_CHECK_RESULT;$HOSTNAME;$SERVICEDESC;$STATUS;$MESSAGE\n" $now > $commandfile

The Nagios or op5 Monitor hosts.cfg

###############################################################################
#  Generated by op5 Monitor webconfiguration exporter
#
#  Exported 2009-10-22 19:33 by monitor
#

# host 'webshopcontainer'
define host{
    use                            Dummy-template
    host_name                      webshopcontainer
    alias                          webshopcontainer
    address                        127.0.0.1
    contact_groups                 support-group
    }
 

The Nagios or op5 Monitor services.cfg

###############################################################################
#  Generated by op5 Monitor webconfiguration exporter
#
#  Exported 2009-10-22 19:33 by monitor
#

# service 'webshopstatus'
define service{
    use                            Dummy-service-template
    host_name                      webshopcontainer
    service_description            webshopstatus
    servicegroups                  webshop
    flap_detection_enabled         0
    }

op5 Monitor is based on Nagios*, the industry standard for enterprise-class IT infrastructure monitoring. Until now, op5 Monitor has had to rely on the basics of standard Nagios Graphical User Interface (GUI) for presentation and overall user experience. With the release of op5 Monitor 4.2 we include a fully functional preview of a brand new GUI, now based on the Ninja Open Source project. The new GUI is based on a modern PHP architecture and is now shipped for real operation testing.

Usability, flexibility and visualization probably stands for more then 80% of overall customer value. Our solutions for full control of our customers IT systems hold a huge amount of both historical data and instant events. Making this information accessible and easy to understand for as many as possible is the single most effective value for our customers. And who is better to give us important feedback on GUI related features and functions then our customers? This is why we now choose to include an early version for test, evaluation and feedback purposes, all to secure the best for the coming official release of Monitor 5.0 later this year, says Jan Josephson, CEO op5 AB.

The new GUI with its flexible interface based on the PHP framework makes it much easier to provide customizations for customers specific needs. With the introduction of widget and support session based authentication it is now possible to fully personalize views and settings. The widget technology further provides a future standard way for two way synchronization and presentation of customer unique data and the presentation of it.

The ability to search, sort and filter vast data volumes has been greatly improved with pagination for views of hosts and services and the possibility to sort by column. The new improved search feature includes auto suggestion and let you search on hosts, services host groups and service groups.
These changes greatly improves the possibilities to answer to customer needs and to develop customized features that has not been possible until now due to the restrictions from the old GUI.
A new module for network visualization (the NagVis project) has been included in op5 Monitor 4.2. You can now visualize your infrastructure using three different map types;

• Auto map, a auto drawn topology map using parent / child relationships.
• Static Map makes it possible to put any object (host, service, host group or service group) on an image of choice.
• Geo map, uses Google maps API, drawing a map by using addresses or gps coordinates on your monitored objects. Object and status information as well as links can be included in the map. The maps can be visualized as widgets on the tactical overview or be used on big monitors in operation centers.
• Largest OSS-project in Network Monitoring, Nagios.

About op5 AB op5 AB is a world leading OSM – Open Source Management – company that delivers software based on open source for control of IT systems and networks. Its main products are op5 Monitor, op5 Statistics and op5 LogServer. The products are based on open source code that op5 further develops, packages, and sells as complete products and systems with services and support. p5 has a large selection of primarily European clients from different market segments for example AGA Linde Gas, The Swedish Customs Service (Tullverket) and other parastatals, several county councils, municipalities and municipal companies. op5 was founded in 2003. The company is based in Sweden with offices in Stockholm and Gothenburg.

www.op5.com

Next week, op5 will release the new Nagios GUI called Ninja. The license is GPL and several people has reported the GUI work perfectly well on vanilla Nagios. Ninja uses a database backend. Merlin, another op5 developed software feeds the Nagios information into the database.

Links:

• op5 -OpenSource based management software company
• Ninja- Nagios Is Now Just Awesome
• Merlin – Module for Effortless Redundancy and Loadbalancing In Nagios
• Try – Ninja GUI

Some screenshoots below:

 Geomap, based on google maps

Tactical overview, with widgets that could be dragged and dropped

Host details

Nagvis integration

By installing wp-cache the testcase went down to 0.4 seconds.

 The drawback is that my counter widget is getting updated every hour instead of every reload of the webpage.

Take a peak at some nice graphs showing that I stopped the increasing loading time and got a speedup from 7.5 seconds to 0.4 seconds:

The plugin used to gather the information is GPLd and developed by op5.

Links

• The article describing howto monitor vmware.
• op5 Monitor – Enterprise class supported monitoring solution based on Nagios.
• Nagios – OpenSource Enterprise class monitoring solution.
• Check_esx plugin created by op5
• op5 – The company behind he plugin.

http://wiki.nagios.org/index.php/SkypeSmsNotification

My recommendation is to use an GSM modem to send SMS, because if the network is down no SMS can be sent with this solution and SMS is normally used as an alternative alarm route.

Another aspect is that Skype is using propertiary protocols and that should be avoided.

I use MythTV quite frequently and noticed that it is instable when using sasc-ng as a decoder to decrypt encrypted DVB-T channels. So approximatly every third day the MythTVbackend server stops and need to be started again. I have wriiten an earlier article about howto monitor MythTV with Nagios or op5 Monitor so I get noticed that it has stopped. But I need to manually start it again. This article describe howto make Nagios or op5 Monitor to start a stopped MythTVbackend. It can be used for starting almost any service.

I have used the examples provided by Ethan at Nagios official documentation describing eventhandlers.

Normally it is not recommended to let a tool like Nagios or op5 Monitor start a service that has stopped, because it is probably a reason why the service has stopped and the correct procedure is to fix the root cause of the problem, not the symptom.

The MythTV backend runs on one machine called lala (after a character in Teletubbies) which is not the same as the Nagios or op5 Monitor server. I use nrpe to run the start script i.e.

 /etc/init.d/mythtv-backend start

There is several options here but I already setup the nrpe agent and it is simple to make Nagios or op5 Monitor to use nrpe to run a script.

Implementation

I used the script I found at Nagios documentation about eventhandlers as a base and modiied it slightly.

At my op5 Monitor machine

/opt/plugins/custom/restart-mythtv-lala.sh

#!/bin/sh
#
# Event handler script for restarting the mythTVbackend server on lala
#
# Note: This script will only restart the mythtvbackend if the service is
#       retried 2 times (in a "soft" state) or if the service somehow
#       manages to fall into a "hard" error state.
#

# What state is the mythbackend service in?
case "$1" in
OK)
	# The service just came back up, so don't do anything...
	;;
WARNING)
	# We don't really care about warning states, since the service is probably still running...
	;;
UNKNOWN)
	# We don't know what might be causing an unknown error, so don't do anything...
	;;
CRITICAL)
	# Aha!  The HTTP service appears to have a problem - perhaps we should restart the server...

	# Is this a "soft" or a "hard" state?
	case "$2" in

	# We're in a "soft" state, meaning that Nagios is in the middle of retrying the
	# check before it turns into a "hard" state and contacts get notified...
	SOFT)

		# What check attempt are we on?  We don't want to restart the web server on the first
		# check, because it may just be a fluke!
		case "$3" in

		# Wait until the check has been tried 3 times before restarting the web server.
		# If the check fails on the 4th time (after we restart the web server), the state
		# type will turn to "hard" and contacts will be notified of the problem.
		# Hopefully this will restart the web server successfully, so the 4th check will
		# result in a "soft" recovery.  If that happens no one gets notified because we
		# fixed the problem!
		2)
			echo "`date` Restarting mythtv service (2rd soft critical state)..." >> /tmp/mythtvstart
			# Call the init script to restart the mythbackend server
			#/etc/rc.d/init.d/httpd restart
			#date >> /tmp/mythtvstart
			/opt/plugins/check_nrpe -H lala -c start_mythtvbackend
			;;
			esac
		;;

	# The mythtvbackend service somehow managed to turn into a hard error without getting fixed.
	# It should have been restarted by the code above, but for some reason it didn't.
	# Let's give it one last try, shall we?
	# Note: Contacts have already been notified of a problem with the service at this
	# point (unless you disabled notifications for this service)
	HARD)
		echo "`date` Restarting mythtv service (hard state)..." >> /tmp/mythtvstart
		# Call the init script to restart the HTTPD server
		#/etc/rc.d/init.d/httpd restart
		#date >> /tmp/mythtvstart
		/opt/plugins/check_nrpe -H lala -c start_mythtvbackend
		;;
	esac
	;;
esac
exit 0

/opt/monitor/misccomands.cfg

# command 'restart-mythtv-lala'
define command{
    command_name                   restart-mythtv-lala
    command_line                   /opt/plugins/custom/start-mythtv-lala.sh $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$
    }

/opt/monitor/etc/services.cfg

# service 'Mythbackend'
define service{
    use                            default-service
    host_name                      lala
    service_description            Mythbackend
    check_command                  check_tcp!6543
    servicegroups                  MythTV,it-slav
    event_handler                  restart-mythtv-lala!$SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$
    contact_groups                 it-slav_sms,it-slav_jabber,it_slav_mail
    }

At my mythbackend machine lala

/etc/nrpe.d/mycommands.cfg
command[start_mythtvbackend]=/usr/bin/sudo /etc/init.d/mythtv-backend start

/etc/sudoers
nobody ALL= (root) NOPASSWD:/etc/init.d/mythtv-backend start

Test

I stopped the mythtvbackend by running:

peter@lala:/etc/nrpe.d$ date
Mon Jun 15 20:40:55 CEST 2009
peter@lala:/etc/nrpe.d$ sudo /etc/init.d/mythtv-backend stop
 * Stopping MythTV server: mythbackend

And run

[root@op5 ~]# tail -f /tmp/mythtvstart
Mon Jun 15 20:47:09 CEST 2009 Restarting mythtv service (2rd soft critical state)...

YES it works!

Links:

• op5 Monitor a Nagios based supported enterprise Monitoring software.
• MythTV a free OpenSource Digital Video Recorder
• Nagios Open Source Monitoring

http://www.op5.org/community/projects/op5-plugins

The plugins is also located in a Git repository and can be checked out from:

http://git.op5.org/git/

Read more at http://www.op5.org/community/projects/merlin

This article describes how to monitor an IPSEC tunnel running on OpenBSD. I could not find any plugin already done so I created my own.

The pre req. for this article are:

• A working Nagios or op5 Monitor setup
• A IPsec VPN tunnel running on OpenBSD
• A working NRPE agent at the OpenBSD box

Theory

The way of getting the status of IPsec on OpenBSD is buy running:

ipsecctl -s s

esp tunnel from x.x.x.x to y.y.y.y spi 0xe58a63d3 auth hmac-md5 enc 3des-cbc \
       authkey 0xabcdfghijklmnopqrstuvxyz \
       enckey 0xabcdfghijklmnopqrstuvxyz
esp tunnel from y.y.y.y to x.x.x.x spi 0x555f1f13 auth hmac-md5 enc 3des-cbc \
      authkey 0xabcdfghijklmnopqrstuvxyz \
       enckey 0xabcdfghijklmnopqrstuvxyz

This shows that the IPsec tunnel between x.x.x.x and y.y.y.y is up.

Depending of the OpenBSD version the output will be different.

Plugin

I put the plugin in /opt/plugins/custom at my OpenBSD box.

#!/bin/sh
#
# Copyright (C) 2009 Peter Andersson, peter@it-slav.net
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
#
# Very simple plugin that checks if a ipsec vpn is up between to ip-adresses
# Tested on OpenBSD 4.0
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see .
#
# Example use of this script:
# ./check_ipsecctl 10.1.1.1 10.2.1.1 "VPN HQ"
# OK: VPN HQ is up
#
# ./check_ipsecctl 10.1.1.1 10.2.1.1 "VPN HQ"
# CRITICAL: VPN HQ is down (No IP-SEC VPN from 10.1.1.1 to 10.2.1.1  No IP-SEC VPN from 10.2.1.1 to 10.1.1.1)
#
#
IPSECCTL="/sbin/ipsecctl -s sa"
STATUS=0

LINE1=`$IPSECCTL | grep "from $1 to $2" `
if [ $? -eq 1 ]; then
        STATUS=2;
        OUTPUT1="No IP-SEC VPN from $1 to $2 "
fi

LINE2=`$IPSECCTL | grep "from $2 to $1" `
if [ $? -eq 1 ]; then
        STATUS=2;
        OUTPUT2="No IP-SEC VPN from $2 to $1"
fi

if [ $STATUS -eq 0 ]; then
        echo "OK: $3 is up"
        exit $STATUS
else
        echo "CRITICAL: $3 is down ($OUTPUT1 $OUTPUT2)"
        exit $STATUS
fi

Nrpe config

Nagios run check_ipsecctl via NRPE, it mus run as a privileged user and I use sudo, in /etc/nrpe.cfg

command[vpn_johan]=sudo /opt/plugins/custom/check_ipsecctl x.x.x.x y.y.y.y "VPN Johan"

x.x.x.x and y.y.y.y are the IP-addresses where the VPN tunnel terminates

Sudo

Use sudoedit /etc/sudoers to modify the sudo config file:

nagios  ALL=(root) NOPASSWD: /opt/plugins/custom/check_ipsecctl

Nagios or op5 Monitor configuration

The VPN connection can be treated as a service running on the OpenBSD box, but in my opinion, the VPN  should be treated as a host using the plugin above to check that the host is alive, and the hosts at the other end of the VPN connection should have the  VPN tunnel as parent. The advantage is that if the VPN tunnel is down the hosts and services behind it is unreachable, which is the correct behavior.

hosts.cfg

# host template 'default-hosttemplate-nrpe'
define host{
    name                           default-hosttemplate-nrpe
    check_command                  check_nrpe
    max_check_attempts             5
    obsess_over_host               0
    check_freshness                0
    active_checks_enabled          1
    passive_checks_enabled         1
    event_handler_enabled          1
    flap_detection_enabled         1
    flap_detection_options         n
    process_perf_data              1
    retain_status_information      1
    retain_nonstatus_information   1
    notification_interval          0
    notification_period            24x7
    notification_options           d,u,r,f
    notifications_enabled          1
    stalking_options               n
    register                       0
    }
# host 'vpn-johan'
define host{
    use                            default-hosttemplate-nrpe
    host_name                      vpn-johan
    alias                          vpn johan
    address                        10.1.1.1
    parents                        internet
    check_command                  check_nrpe!vpn_johan
    contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
    }

10.1.1.1 is the IP-adress to my OpenBSD box. The reason for using a template is that I’m using the webbased config tool that comes with op5 Monitor.

The result

Links

• Nagios
• op5 Monitor a Nagios based full supported monitor solution
• OpenBSD, a FREE, multi-platform 4.4BSD-based UNIX-like operating system.
• IPsec, a suite of protocols for securing Internet Protocol communications by authenticating and encrypting each IP packet of a data stream.

The demo is hosted by Collax a op5 partner in Germany.

If you want to attend, register at:

http://www.collax.com/de/ueber-collax/events-collax-live/webcast-collax-monitoring-solution.html

This article describe howto use a USB camera (also known as webcam) to detect motions and send an alarm to Nagios or op5 Monitor. It can be used in datacenters to send an alert if someone or something is moving or at home to detect if someone is in your house. I will place the camera just inside the door so everybody passing my front door will be filmed and an alert will be sent to me.

The software used:

• Motion, to detect movement and create a movie of the relevant time, pictures is also taken.
• Nagios or op5 Monitor to create the alarm.

The hardware used is an old Logitech webcam I found in my fathers drawer Motion supports alot of different devices but must simple is to use a video4linux supported device.

The pre req. for this article is a working Nagios or op5 Monitor system.

Installation

I have an Ubuntu system where I attach the webcamera, dmesg gives me:

[634581.861029] usb 1-2: new full speed USB device using uhci_hcd and address 3
[634582.032251] usb 1-2: configuration #1 chosen from 1 choice
[634582.034125] quickcam: QuickCam USB camera found (driver version QuickCam USB 0.6.6 $Date: 2006/11/04 08:38:14 $)
[634582.034144] quickcam: Kernel:2.6.27-11-generic bus:1 class:FF subclass:FF vendor:046D product:0870
[634582.048202] quickcam: Sensor HDCS-1020 detected
[634582.050173] quickcam: Registered device: /dev/video0

Installing motion is simple, just run:

sudo apt-get install motion

I will use passive checks to let motion send in alerts to Nagios, the best way of doing that is by using NSCA, for theory refer to an earlier article.

sudo apt-get install nsca

Test motion

The first thing to test is that motion works, for me to get it to work I needed no configuration, run:

sudo motion

and dance in front of your camera.

Now motion should inform that it has created pictures and a movie.

Configuration

Configure Nagios or op5 Monitor to accept alerts sent from motion:

# service 'Camera Motion Detection'
define service{
 use                            default-service
 host_name                      lala
 service_description            Camera Motion Detection
 check_command                  check_dummy!0 "No motion detected"
 max_check_attempts             1
 active_checks_enabled          0
 flap_detection_options         n
 contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
 stalking_options               n
 check_freshness                1
 freshness_threshold            3600
}

If nothing has been sent within an hour (3600 seconds), the state is set to OK by using freshness and check_dummy.

Create /home/motion/scripts/check_camera_motion/send_nsca.cfg

encryption_method=0

Create /home/motion/scripts/check_camera_motion/passive_camera_warning

lala    Camera Motion Detection    1    WARNING: Motion detected

The separator must be TAB

In /etc/motion/motion.conf, find the parameter on_event_start and change it to:

on_event_start  cd /home/motion/scripts/check_camera&&send_nsca -H op5.mynet -c send_nsca.cfg < passive_camera_warning

And an alert will be sent to Nagios everytime motion detects a motion.

Start motion with:

sudo motion

And do the dancing again, now you should have an Nagios alarm with status WARNING.

Hint

A good idea could be to change the parameters target_dir and snapshot_filename to a directory where you can point apache websever, so you can use a webbrowser to browse your images and movies.

Links

• Motion, the software used to detect motions
• Nagios
• op5 Monitor, a supported version of Nagios

Pre requirements

The pre requriements for this article is a working Nagios or op5 Installation and a working MySql database.

Plugin

The plugin used in this article is: http://www.consol.de/opensource/nagios/check-mysql-health

Unfortunatly the webpage is in German only.

Enable MySql to retrive information

mysql> grant usage on *.* to 'nagios'@'op5' identified by 'XXXX-';

op5 is the nagios hostname.

Download the plugin, install and test

Download the plugin from http://www.consol.de/fileadmin/opensource/Nagios/check_mysql_health-2.0.4.tar.gz

I put it into /opt/plugins/custom/check_mysql_health at the Nagios server.

Test the command by running at the Nagios server:

[root@op5 custom]# ./check_mysql_health -H mysqlhost --user nagios --password XXXX  --mode threads-connected
 OK - 5 client connection threads | threads_connected=5;10;20

It works!!

To get a list of what the check_mysql_health plugin can monitor run:

[root@op5 custom]# ./check_mysql_health --help

Configure Nagios or op5 Monitor

I find the following parameters interesting to monitor:

• connection-time
• keycache-hitrate
• qcache-hitrate
• tablecache-hitrate
• threadcache-hitrate
• tmp-disk-tables

checkcommands.cfg

# command 'check_mysql_health'
 define command{
 command_name                   check_mysql_health
 command_line                   $USER1$/custom/check_mysql_health -H $HOSTADDRESS$ --user $ARG1$ -password $ARG2$ --mode $ARG3$
 }

# command 'check_mysql_health_tresholds'
 define command{
 command_name                   check_mysql_health_tresholds
 command_line                   $USER1$/custom/check_mysql_health -H $HOSTADDRESS$ --user $ARG1$ -password $ARG2$ --mode $ARG3$ --warning $ARG4$ --critical $ARG5$
 }

I created a servicegroup named mysql

servicegroups.cfg

# servicegroup 'mysql'
 define servicegroup{
 servicegroup_name              mysql
 alias                          MySQL
 }

I use the default thresholds in my servicedefinitions, but use check_mysql_health_thresholds instead if thresholds are required.
services.cfg

# service 'MySQL connection-time'
 define service{
 use                            default-service
 host_name                      lala
 service_description            MySQL connection-time
 check_command                  check_mysql_health!nagios!XXXX!connection-time
 servicegroups                  mysql
 contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
 }

# service 'MySQL keycache-hitrate'
 define service{
 use                            default-service
 host_name                      lala
 service_description            MySQL keycache-hitrate
 check_command                  check_mysql_health!XXXX!keycache-hitrate
 servicegroups                  mysql
 contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
 }

# service 'MySQL qcache-hitrate'
 define service{
 use                            default-service
 host_name                      lala
 service_description            MySQL qcache-hitrate
 check_command                  check_mysql_health!nagios!XXXX!qcache-hitrate
 servicegroups                  mysql
 contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
 }

# service 'MySQL tablecache-hitrate'
 define service{
 use                            default-service
 host_name                      lala
 service_description            MySQL tablecache-hitrate
 check_command                  check_mysql_health!nagios!XXXX!tablecache-hitrate
 servicegroups                  mysql
 contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
 }

# service 'MySQL threadcache-hitrate'
 define service{
 use                            default-service
 host_name                      lala
 service_description            MySQL threadcache-hitrate
 check_command                  check_mysql_health!nagios!XXXX!threadcache-hitrate
 servicegroups                  mysql
 contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
 }

# service 'MySQL tmp-disk-tables'
 define service{
 use                            default-service
 host_name                      lala
 service_description            MySQL tmp-disk-tables
 check_command                  check_mysql_health!nagios!XXXX!tmp-disk-tables
 servicegroups                  mysql
 contact_groups                 it-slav_msn,it-slav_mail,call_it-slav
 }

The result

Conclusions and wrap-up

The default values are in many cases not sufficent, I have no database that do not get WARNING or CRITICAL when the default thresholds are used. If that is because the thresholds are not realistic or if it my applications that are badly implemented I do not know. So the databases needs tuning and the thresholds must be changed to get rid of constant errormessages.

Links

• op5 Monitor
• Nagios
• Tuning MySQL
• The plugin used in this article check_mysql_health

Wordpress uses the following parts to work:

• Webserver frontend
• PHP scripts
• MySQL database backend
• At least one system to run it on

I found one article describing howto monitor the Webserver and the MySQL database and even that a certain webpage contain a certain phrase. But I would like to emulate a user that clicks on some pages and graph the performance.

The tool I like for in-depth web monitoring is webinject. It allows scripting a simulated web user with alot of features.

I recommend to read the excellent how-to written by op5 support team and of cource use webinjects reference manual.

The testcase I’ve created looks like this:

<testcases repeat="1">
<case
    id="1"
    description1="Startsidan www.it-slav.net/blogs"
    method="get"
    url="http://www.it-slav.net/blogs/"
    verifypositive="An It-Slave in the digital saltmine"
/>
<case
    id="2"
    description1="Nagios or op5 Monitor plugin for 1-wire humidity measurement"
    method="get"
    url="http://www.it-slav.net/blogs/?p=802"
    verifypositive="1-wire humidity probe"
/>
<case
    id="3"
    description1="op5 Monitor or Nagios plugin for 1-wire temperature measurement"
    method="get"
    url="http://www.it-slav.net/blogs/?p=115"
    verifypositive="check_1-wiretemp"
/>
<case
    id="4"
    description1="Wp-Admin"
    method="get"
    url="http://www.it-slav.net/blogs/wp-admin"
    verifyresponsecode="301"
/>
</testcases>

And when run in Nagios mode:

[root@op5 webinject]# ./webinject.pl -c config.xml testcases/wordpressblog.xml
WebInject OK - All tests passed successfully in 4.143 seconds |time=4.143;10;;0

In checkcommands.cfg:

# command 'check_webinject'
define command{
    command_name                   check_webinject
    command_line                   $USER1$/webinject/webinject.pl testcases/$ARG1$
    }

In services.cfg:

# service 'Webinject Blog'
define service{
    use                            default-service
    host_name                      dull
    service_description            Webinject Blog
    check_command                  check_webinject!wordpressblog.xml
    contact_groups                 it-slav_sms,it-slav_mail,call_it-slav
    }

This is a nice pnp graph of my Webinject check:

Links:

• Webinject
• op5 Webinject Howto
• Nagios
• op5 Monitor