It really annoyed me that the iowait at the cpu on the Dom 0 system was approximatly 50% when the 4 virtual system was doing more or less nothing.

I showed my earlier blogpost about VM virtualization to our sysadmin at op5. He gave me a couple of more hints that really make a big difference, the best part is that they are real simple to implement.

I also noticed that I forgot some parts in the previous article that was obvius to me.

#1 use virtio drivers if possible

This is probably the single most important step to get the most out of the hardware.

When the virtual systems wants to communicate with the hardware it uses drivers, if the drivers are simulating hardware to give a service to the virtual operating system it adds alot of overhed. A simple example:

• The virtual system wants to access the disk
• The virtual system communicates via the driver to the hardware
• The hardware is not real hardware, it is emulated hardware in sofware 
• The software that emulates the hardware sends the I/O to the Dom 0 virtualization engine
• The Virtualization engine verifies that the access is ok, translate it to real hardware call via its driver
• The hardware do the operation and send the result back up via all the steps to the virtual operating system

All this transalations and software calls take alot of resources from the machine.

By using virtual drivers, so that there is no need to emulate hardware, many of the steps can be skipped or done with much less penalty. However not all operating system have virtio drivers.

#2 load the virtio modules in Dom 0

This is a real god hint I got from our sysadmin, load the virtual drivers in the Dom 0 system asweell. According to him, the virtual drivers in the Dom 0 and Dom U:s communicate and makes the I/O much more effective. I have not found any information about this at any place so I was sceptical but I tried it.

# modprobe virtio_balloon
# modprobe virtio_blk
# modprobe virtio_net
# modprobe virtio_pci

The virtual systems need to be rebooted as far as I know before the loaded drivers will have any effect.

At first no change at all, but after few hours I noticed that CPU idle raized and the iowait went from at least 50% down to below 5%.

A nice graph, created by op5 Monitor showing the difference:

I loaded the virtual drivers approximately 4 pm, and it took until 10 pm before the performance boost showed up.

The correct way of making sure that the drivers are loaded directly would be to remake the kernel but I'm lazy so I have just added the module load to /etc/rc.local

Conclusions

It is always important to monitor systems. When doing changes it is not possible know if the changes do have any impact if you do not monitor and follow up how and if the changes does anything.

My former statement that KVM is not mature is more true then I have thought. The knowledge how to tune and maintain KVM systems seems to be like black art where the knowledge is not documented and well spread. I have read the RedHat virtualization and I have not found anything about that the virtual drivers should be loaded on the Dom0 system.

If there is anyone out there that has some documentation or more hints, please send me the link or comment this blogpost.

Links:

• RedHat virtualization doc
• op5 Monitor that monitor and creates nice graphs
• check_cpu_stats Nagios plugin to monitor cpustat

As an it slave, I have the same problems as most datacenters:

• Running out of space because of more and more machines get into the datacenter
• Overheating
• Powerconsumption

Added to the "normal" datacenter problems, I also have to deal with:

• Wife acceptance
• Noice, my neighbours has complained about the noice outside my basement.

To address this issues I will run a consolidation project to get fewer machines by using virtualization. As virtualization engine, KVM is choosed.

This article will describe some pitfalls I run into and how I solved them.

KVM plattform

KVM seems to be the future for virtualization within the Linux area since Xen has done some mistakes.

As I have good experience of running CentOS and has KVM included I decided to use CentOS as the plattform for my virtualization.

Issues

Bridge network

To let the virtual boxes has full access to the network you need create a network bridge, it is rather straightforward and well documented so I will skip this part. I used the documentation on Red Hat customer support. Why KVM requires a bridge instead of a normal NIC is another question

Snapshots

One handy feature with virtualization is that the virtual systems can be one big fat file at the physical host, this gives the possibility to make full backups without turning the system down a.k.a snapshots.

However to get snapshot to work the filetype must be qcow2 and raw is the default format.

It is possible to convert from raw to qcow2 format by running the command:

qemu-img convert -f raw -O qcow2 <virtualhost>.img <virtualhost>.img.qcow2

Remember to turn of your virtual system by running:

virsh shutdown <virtualhost>

After the conversion:

virsh edit <virtualhost>

modify the type and path, i.e.

      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/web.img.qcow2'/>

Start your virtual system:

virsh start <virtualhost>

Take a snapshot with:

qemu-img snapshot -c <snapshotname> <virtualhost>.img.preallc.qcow2

To create a full image of your snapshot you need to compile a more resent version of qemu-img then shipped with CentOS 6.2, I download a later version and compiled it from here, i renamed the new to qemu.img2 and copied it to /usr/local/bin

qemu-img2 convert -p -f qcow2 -O qcow2 -s <snapshotname> <virtualhost>.img.preallc.qcow2 <targetpath>

Beware: This command can use all your resources and may affect all your virtual systems, use nice and ionice to prevent it.

Disk I/O

After installing just a few virtual system I noticed that my physical host got alot of I/O wait, it can be seen by using tools like top.

To monitor this I installed the nagios plugins:

• check disk io, can be found here
• check cpu stats, can be found here
• check_libvirt, developed by op5 and an Howto

However it seems like that though my virtual systems did more or less nothing they caused alot of disk I/O on the physical system.

I did some investigations and read quite many articles, fiddled and tested. The following is what worked for me in my setup.

#1 Change disk scheduler

The disk scheduler can be changed on runtime by modify the file:

/sys/block/sda/queue/scheduler

to see what scheduler you use now:

cat /sys/block/sda/queue/scheduler

Change sda to the device you have.

The scheduler that worked best for me is deadline

echo deadline > /sys/block/sda/queue/scheduler

#2 Mount with noatime

A feature in Unix and other Unixlike systems like Linux is that it normaly stores when a file is accessed. So one read always produce a write and if you are using raid like morroring this get worse, one reade always generate several writes. This creates alot of overhead for a feature soldom used.

So change /etc/fstab so it will mount the file systems with noatime.
An axample:

UUID=a290aa4b-635c-45fa-b144-1fbef90b3735 /                       ext4    defaults,noatime        1 1

#3 Preallocation disk images

A real boost that is hidden in the featureset and not shown in the virtualmachine gui is using preallocation disk images. They cannot be created from the GUI so I install the virtual machine, turn it of and convert it afterwards.

qemu-img convert -f qcow2 -O qcow2 -o preallocation=metadata <virtualhost>.img.qcow2 <virtualhost>.img.preallc.qcow2

Change the path to new image name by editing the virtual machine settings

virsh edit <virtualhost>

Conclusions

In my opinion it seems like KVM is still a little bit immature or at least the tools to handle it. Maybe it would be a good idea to have one linux distro focusing on beeing the best platform for virtualization.

I am convinced that I can get even more bang for the bucks out of my installation if I learn how to tweak it even more, so if you have any hints, do not hesitate to contact me. Preferably as a comment to this blogpost.

Referenses:

• op5 Monitor, a nagios based enterprise monitor tool to monitor my environment
• check_libvirt, a nagios plugin done by op5 to monitor KVM
• check_diskio, a nagios plugin to monitor disk I/O
• check_cpu_stats, a nagios plugin to monitor cpu usage rather detailed
• Another blogpost which helped me
• KVM performance improvements and optimizations – Red Hat presentation

• Support for offline maps – no need to have online gmap connection
• Introducing IP traffic “weathermaps” i.e. a user friendly way to present traffic flows, being WAN or LAN traffic
• Display monitor data on external web i.e visualise and distribute the information to greater crowd
• Multiple of  time saving tasks such as multidelete, free text filter, save searches etc.

Having a modern easy to use proactive monitoring system is the core of assuring quality delivery of your enterprise applications to your user. However cool report and dashboards are great, if they require complex skills, many hours to set up and maintain -  the value is quickly gone.

With op5 Monitor v5.5 we introduce a number of features to make the information more presentable and understandable. We have made lots of new micro features to make the everyday change management easier and faster.

Major features in this release:   
•    Support for offline map
    ⁃    Now integrating the open streetmap (OSM) project, making the vizualisation of large networks easier and offline
    •    Support for IP traffic flow visualisation on maps or any background
    •    Publish op5 Monitor data / widget on external web service
    •    Multi tier host & service groups
    ⁃    A host or service group can now be part of another group creating a tree structure of dependencies.
    •    Multidelete – it’s always easy to add a new host or service to a system – now it´s easy to remove as well – saves lots of time!
Usability improvements:
    •    Saved searches
    •    Search on notes and comments and search in scheduling queue
    •    Free text filter
    •    Enhanced Alert history
    •    Authentication fallback

In total we have fixed and added more then 240 items in this release. Please let us know what you think – we focus on Your needs, just tell us!

Best regards
The op5 Team!

More about the new release

My old NAS built on top of CentOS has some performence issues and the disks are quit old now. The hardware is leftovers that is approx 8 years old and not highend at that time either. The disks age has worried my for a time and I would like to have higher redunduncy with raid1/mirrored disks.

After some research I decided to run my new NAS on FreeNAS, which has all the features I use, like: CIFS, NFS, FTP, SSH, Rsync, Unison, iSCSI and also easy administration and disk management.

This article will describe how I did setup Nagios or op5 Monitoring on my FreeNAS system. I assume basic knowledge of Nagios or op5 Monitor.

Installing nrpe

Nrpe is the agent that nagios and op5 Monitor uses to monitor a system from the inside.

From the webgui add group nagios and user nagios. User nagios should be a meber of group nagios.

 freenas:~# pkg_add -r nrpe2
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/Latest/nrpe2.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/perl-5.10.1.tbz... Done.
Removing stale symlinks from /usr/bin...
    Skipping /usr/bin/perl
    Skipping /usr/bin/perl5
Done.
Creating various symlinks in /usr/bin...
    Symlinking /usr/local/bin/perl5.10.1 to /usr/bin/perl
    Symlinking /usr/local/bin/perl5.10.1 to /usr/bin/perl5
Done.
cd: can't cd to /usr/include
Cleaning up /etc/make.conf... Done.
Spamming /etc/make.conf... Done.
Cleaning up /etc/manpath.config... Done.
Spamming /etc/manpath.config... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/libiconv-1.13.1_1.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/gettext-0.17_1.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.3-release/All/nagios-plugins-1.4.14,1.tbz... Done.
You already have a "nagios" group, so I will use it.
You already have a "nagios" user, so I will use it.

**********************************************************************
 

freenas:~# cp /usr/local/etc/nrpe.cfg-sample /etc/nrpe.cfg

I changed the allowed_hosts line and added my op5 Monitor host.

So the nagios agent works.

Links

• op5, an Enterprise Monitor product company
• op5 Monitor, Enterprise Monitoring based on opensource
• Nagios, an excellent opensource monitor tool
• FreeNAS, very capable NAS based on FreeBSD

The widget keeps track of how many times the word "moln" appear on idg.se webpage and displays it in a easy view in The Tactical Overview.

Screenshoots from the different modes:

Links:

• op5 Sandbox with a live cloud widget online, registration is necessary
• Ninja, the new and inovative nagios gui
• Widget exchange area

Kudos to op5 support that has created this excellent widget 

The dashboard has been modified and improved by the community and an example can be seen at op5 sandbox dashboard and can be downloaded from op5 community.

The original project place is http://dingleberry.me/2010/04/our-new-dashboard/

When I was preparing a presentation about what op5 is doing and our contribution to the community, I went to ideas.nagios.org. When I browsed the list of the biggest issues with Nagios I found out that op5 has packaged and solved them all in op5 Monitor. I encourage everyone to take a peak at the list and judge for them self what platform you want to use for your enterprise monitoring solution.

1. Nagios Clusters

op5 has developed Merlin (Module for Effortless and Redundant and Loadbalanced Infrastructure with Nagios). It provides the possibility to have a redundant or/and loadbalanced solution using Nagios. It also provides a scalable database backend for Ninja and Nagvis, take a peak at Merlin webpage.

2. Performance Graphing

In op5 Monitor op5 has added pnp4nagios that graphs more or less any check that gives a numerical value back.

3. New Status Map

We provide a new statusmap, it is a part of Nagvis project and called Automap. op5 has contributed to Nagvis by adding support for Merlin database backend and GoogleMaps integration

4. Better Interface

op5 has created a new PHP based interface that makes you proud of your monitor solution, called Ninja. Ninja stands for Nagios is now Just Awesome, it looks so good so you can show it to your manager:-) Take a peak at Ninja webpage.

We had a naming competition at op5 my suggestion to call it Yang, Yet Another Nagios Gui, did not get so many votes so the name become Ninja.

5. Configuration

Any one that has tried to configure Nagios by hand has thought bad thoughts. It is cumbersome and in the begining it makes even a skilled Unix admin frustrated. op5 has developed a web based configuration tool named Nacoma. It is included in op5 Monitor.

6. SLA reports

op5 has got alot of feedback from the customers and one issue was better reports, we have created two reports: availability and SLA report. Both of them can be scheduled and looks god compared to the reports in Nagios, they are included in Ninja

 
7. More members in core team

Well, this is something that op5 cannot controll, however, Andreas Ericsson one of op5s core developers are a member og Nagios steering board.

8. SNMP Trap receiver

This is something op5 is working on right now, we have today created a prototype together with a customer and depending of the outcome of that project we will know better when it will be available to the market.

9. UI-Improvment

op5s solution to this issue is Ninja, a better, nicer, scalable, database backend gui to Nagios.

10. Nagios Dashboard

A newTactical Overview with widgets or Nagvis, all included in Ninja

Links

• ideas.nagios.org http://ideas.nagios.org
• Nagios http://www.nagios.org
• op5 http://www.op5.com
• op5 Monitor http://www.op5.com/op5/products/network-monitor
• Ninja http://www.op5.org/community/plugin-inventory/op5-projects/ninja
• Merlin http://www.op5.org/community/plugin-inventory/op5-projects/merlin
• Nagvis http://www.nagvis.org/
• pnp4nagios http://docs.pnp4nagios.org/

I have registred me to attend the open Splunk event in Stockholm. Within 24 hours I got an email telling that they do not want competitors at the event and would like to discuss the purpose of me attending.

I called the Splunk representative and told him that op5 thinks Splunk is a competitor in some cases and in some cases we have products that run in parallel at customer sites. We want to integrate op5 Monitor with Splunk because we have got the question a couple of times from customers and potential customers. My intention of attending the meeting is to get a better understanding of Splunk so op5 can integrate our products, so our customers get the products that suits them best. If Splunk wants to be part of the integration, the better.

The Splunk representative still do not want me to attend. At Splunk they will discuss this matter internally but so far, this is their view.

I would really like to make a parallel with another op5 competitor, Würth, at their event they invited op5 as key speaker and I really recommend Splunk to reconsider their decision.

• op5 Monitor Beta, virtual image in ovf format
• Try op5 Monitor Beta

The op5 Monitor Beta use Ninja as GUI and Merlin as a databasebackend.

–cut–

• Implementation of the missing Nagios cgi’s. (showlog, history,trends, notifications etc)
• Include Avail/SLA reports in Ninja  (Avail reports are an improved version of the Nagios avail reports and SLA is a new report type)
• Lots of bugfixes

–cut–

Links

• Ninja, Nagios Is Now Just Awesome
• Merlin, Module for Endless Redundacy and Loadbalancing In Nagios
• op5, a company with enterprise network- and systems management based on OpenSource.

I was reading the Swedish computer magazine, Tech World. There was an article about an appliance solution that could give a report about unused switch ports. The box PortIQ from Infoblox costs approximatly 10.000 Euro.

op5 has developed this feature at a customer site and is included with op5 Monitor, op5 has published the code as opensource.

This guide will describe howto get it running in op5 Monitor. op5 Monitor is vanilla Nagios with some addons so it should be no problem to follow this guide in a Nagios environment

Installation

1. Update your op5 Monitor or download the plugin and report script.

2. Create metadata for check_portstatus as $USER1$/check_portstatus -H $HOSTADDRESS$

in checkcommands.cfg:

# command 'check_portstatus'
define command{
    command_name                   check_portstatus
    command_line                   $USER1$/check_portstatus -H $HOSTADDRESS$
    }
 

3. mkdir /var/spool/check_portstatus

4. chown monitor:apache /var/spool/check_portstatus

5. cp /opt/monitor/op5/portstatus/portstatus.php /var/www/html/portstatus.php

6. Define your service i.e. services.cfg:

# service 'Portstatus'
define service{
    use                            default-service
    host_name                      gw
    service_description            Portstatus
    check_command                  check_portstatus
    }
 

The result

The screenshot below shows a Cisco switch with alot of unused ports.

Conclusions

This article shows how easy it is to modify Nagios and other opensource tools to give you what you want. Do not hesitate to contact me or op5 to help you enhance you Nagios or op5 installation.

Links

• op5
• op5 Monitor
• Nagios
• PortIQ appliance box from Infoblox
• Tech world

op5 Monitor is based on Nagios*, the industry standard for enterprise-class IT infrastructure monitoring. Until now, op5 Monitor has had to rely on the basics of standard Nagios Graphical User Interface (GUI) for presentation and overall user experience. With the release of op5 Monitor 4.2 we include a fully functional preview of a brand new GUI, now based on the Ninja Open Source project. The new GUI is based on a modern PHP architecture and is now shipped for real operation testing.

Usability, flexibility and visualization probably stands for more then 80% of overall customer value. Our solutions for full control of our customers IT systems hold a huge amount of both historical data and instant events. Making this information accessible and easy to understand for as many as possible is the single most effective value for our customers. And who is better to give us important feedback on GUI related features and functions then our customers? This is why we now choose to include an early version for test, evaluation and feedback purposes, all to secure the best for the coming official release of Monitor 5.0 later this year, says Jan Josephson, CEO op5 AB.

The new GUI with its flexible interface based on the PHP framework makes it much easier to provide customizations for customers specific needs. With the introduction of widget and support session based authentication it is now possible to fully personalize views and settings. The widget technology further provides a future standard way for two way synchronization and presentation of customer unique data and the presentation of it.

The ability to search, sort and filter vast data volumes has been greatly improved with pagination for views of hosts and services and the possibility to sort by column. The new improved search feature includes auto suggestion and let you search on hosts, services host groups and service groups.
These changes greatly improves the possibilities to answer to customer needs and to develop customized features that has not been possible until now due to the restrictions from the old GUI.
A new module for network visualization (the NagVis project) has been included in op5 Monitor 4.2. You can now visualize your infrastructure using three different map types;

• Auto map, a auto drawn topology map using parent / child relationships.
• Static Map makes it possible to put any object (host, service, host group or service group) on an image of choice.
• Geo map, uses Google maps API, drawing a map by using addresses or gps coordinates on your monitored objects. Object and status information as well as links can be included in the map. The maps can be visualized as widgets on the tactical overview or be used on big monitors in operation centers.
• Largest OSS-project in Network Monitoring, Nagios.

About op5 AB op5 AB is a world leading OSM – Open Source Management – company that delivers software based on open source for control of IT systems and networks. Its main products are op5 Monitor, op5 Statistics and op5 LogServer. The products are based on open source code that op5 further develops, packages, and sells as complete products and systems with services and support. p5 has a large selection of primarily European clients from different market segments for example AGA Linde Gas, The Swedish Customs Service (Tullverket) and other parastatals, several county councils, municipalities and municipal companies. op5 was founded in 2003. The company is based in Sweden with offices in Stockholm and Gothenburg.

www.op5.com

Next week, op5 will release the new Nagios GUI called Ninja. The license is GPL and several people has reported the GUI work perfectly well on vanilla Nagios. Ninja uses a database backend. Merlin, another op5 developed software feeds the Nagios information into the database.

Links:

• op5 -OpenSource based management software company
• Ninja- Nagios Is Now Just Awesome
• Merlin – Module for Effortless Redundancy and Loadbalancing In Nagios
• Try – Ninja GUI

Some screenshoots below:

 Geomap, based on google maps

Tactical overview, with widgets that could be dragged and dropped

Host details

Nagvis integration

The plugin used to gather the information is GPLd and developed by op5.

Links

• The article describing howto monitor vmware.
• op5 Monitor – Enterprise class supported monitoring solution based on Nagios.
• Nagios – OpenSource Enterprise class monitoring solution.
• Check_esx plugin created by op5
• op5 – The company behind he plugin.

http://www.op5.org/community/projects/op5-plugins

The plugins is also located in a Git repository and can be checked out from:

http://git.op5.org/git/

• Tobias Oitiker gave a very good presentation about the linux kernel caching and described how it affected RRDtools.
• Markus Almroth, describes Sysnagios which did the opposite of most monitoring projects. Sysnagios decentralized the configuration of systems and network monitoring in a controlled way. I will come back to this topic in my blog.
• Per Åsberg talked about Ninja, a new GUI for Nagios, done by op5.
• Andriy Skuratov, talked about and demonstrated a new version of Nagvis with integrated Google Maps and other good stuff.
• Ethan Galstad the founder of Nagios announced Ninja as the new GUI for Nagios.

During the Nordic meet on Nagios I meet alot of interesting people, learned alot and had good food and drinks.

I hope to see you next year at Nordic Meet on Nagios 2010!!!

Read more at http://www.op5.org/community/projects/merlin

The demo is hosted by Collax a op5 partner in Germany.

If you want to attend, register at:

http://www.collax.com/de/ueber-collax/events-collax-live/webcast-collax-monitoring-solution.html