It really annoyed me that the iowait at the cpu on the Dom 0 system was approximatly 50% when the 4 virtual system was doing more or less nothing.

I showed my earlier blogpost about VM virtualization to our sysadmin at op5. He gave me a couple of more hints that really make a big difference, the best part is that they are real simple to implement.

I also noticed that I forgot some parts in the previous article that was obvius to me.

#1 use virtio drivers if possible

This is probably the single most important step to get the most out of the hardware.

When the virtual systems wants to communicate with the hardware it uses drivers, if the drivers are simulating hardware to give a service to the virtual operating system it adds alot of overhed. A simple example:

• The virtual system wants to access the disk
• The virtual system communicates via the driver to the hardware
• The hardware is not real hardware, it is emulated hardware in sofware 
• The software that emulates the hardware sends the I/O to the Dom 0 virtualization engine
• The Virtualization engine verifies that the access is ok, translate it to real hardware call via its driver
• The hardware do the operation and send the result back up via all the steps to the virtual operating system

All this transalations and software calls take alot of resources from the machine.

By using virtual drivers, so that there is no need to emulate hardware, many of the steps can be skipped or done with much less penalty. However not all operating system have virtio drivers.

#2 load the virtio modules in Dom 0

This is a real god hint I got from our sysadmin, load the virtual drivers in the Dom 0 system asweell. According to him, the virtual drivers in the Dom 0 and Dom U:s communicate and makes the I/O much more effective. I have not found any information about this at any place so I was sceptical but I tried it.

# modprobe virtio_balloon
# modprobe virtio_blk
# modprobe virtio_net
# modprobe virtio_pci

The virtual systems need to be rebooted as far as I know before the loaded drivers will have any effect.

At first no change at all, but after few hours I noticed that CPU idle raized and the iowait went from at least 50% down to below 5%.

A nice graph, created by op5 Monitor showing the difference:

I loaded the virtual drivers approximately 4 pm, and it took until 10 pm before the performance boost showed up.

The correct way of making sure that the drivers are loaded directly would be to remake the kernel but I'm lazy so I have just added the module load to /etc/rc.local

Conclusions

It is always important to monitor systems. When doing changes it is not possible know if the changes do have any impact if you do not monitor and follow up how and if the changes does anything.

My former statement that KVM is not mature is more true then I have thought. The knowledge how to tune and maintain KVM systems seems to be like black art where the knowledge is not documented and well spread. I have read the RedHat virtualization and I have not found anything about that the virtual drivers should be loaded on the Dom0 system.

If there is anyone out there that has some documentation or more hints, please send me the link or comment this blogpost.

Links:

• RedHat virtualization doc
• op5 Monitor that monitor and creates nice graphs
• check_cpu_stats Nagios plugin to monitor cpustat

As an it slave, I have the same problems as most datacenters:

• Running out of space because of more and more machines get into the datacenter
• Overheating
• Powerconsumption

Added to the "normal" datacenter problems, I also have to deal with:

• Wife acceptance
• Noice, my neighbours has complained about the noice outside my basement.

To address this issues I will run a consolidation project to get fewer machines by using virtualization. As virtualization engine, KVM is choosed.

This article will describe some pitfalls I run into and how I solved them.

KVM plattform

KVM seems to be the future for virtualization within the Linux area since Xen has done some mistakes.

As I have good experience of running CentOS and has KVM included I decided to use CentOS as the plattform for my virtualization.

Issues

Bridge network

To let the virtual boxes has full access to the network you need create a network bridge, it is rather straightforward and well documented so I will skip this part. I used the documentation on Red Hat customer support. Why KVM requires a bridge instead of a normal NIC is another question

Snapshots

One handy feature with virtualization is that the virtual systems can be one big fat file at the physical host, this gives the possibility to make full backups without turning the system down a.k.a snapshots.

However to get snapshot to work the filetype must be qcow2 and raw is the default format.

It is possible to convert from raw to qcow2 format by running the command:

qemu-img convert -f raw -O qcow2 <virtualhost>.img <virtualhost>.img.qcow2

Remember to turn of your virtual system by running:

virsh shutdown <virtualhost>

After the conversion:

virsh edit <virtualhost>

modify the type and path, i.e.

      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/web.img.qcow2'/>

Start your virtual system:

virsh start <virtualhost>

Take a snapshot with:

qemu-img snapshot -c <snapshotname> <virtualhost>.img.preallc.qcow2

To create a full image of your snapshot you need to compile a more resent version of qemu-img then shipped with CentOS 6.2, I download a later version and compiled it from here, i renamed the new to qemu.img2 and copied it to /usr/local/bin

qemu-img2 convert -p -f qcow2 -O qcow2 -s <snapshotname> <virtualhost>.img.preallc.qcow2 <targetpath>

Beware: This command can use all your resources and may affect all your virtual systems, use nice and ionice to prevent it.

Disk I/O

After installing just a few virtual system I noticed that my physical host got alot of I/O wait, it can be seen by using tools like top.

To monitor this I installed the nagios plugins:

• check disk io, can be found here
• check cpu stats, can be found here
• check_libvirt, developed by op5 and an Howto

However it seems like that though my virtual systems did more or less nothing they caused alot of disk I/O on the physical system.

I did some investigations and read quite many articles, fiddled and tested. The following is what worked for me in my setup.

#1 Change disk scheduler

The disk scheduler can be changed on runtime by modify the file:

/sys/block/sda/queue/scheduler

to see what scheduler you use now:

cat /sys/block/sda/queue/scheduler

Change sda to the device you have.

The scheduler that worked best for me is deadline

echo deadline > /sys/block/sda/queue/scheduler

#2 Mount with noatime

A feature in Unix and other Unixlike systems like Linux is that it normaly stores when a file is accessed. So one read always produce a write and if you are using raid like morroring this get worse, one reade always generate several writes. This creates alot of overhead for a feature soldom used.

So change /etc/fstab so it will mount the file systems with noatime.
An axample:

UUID=a290aa4b-635c-45fa-b144-1fbef90b3735 /                       ext4    defaults,noatime        1 1

#3 Preallocation disk images

A real boost that is hidden in the featureset and not shown in the virtualmachine gui is using preallocation disk images. They cannot be created from the GUI so I install the virtual machine, turn it of and convert it afterwards.

qemu-img convert -f qcow2 -O qcow2 -o preallocation=metadata <virtualhost>.img.qcow2 <virtualhost>.img.preallc.qcow2

Change the path to new image name by editing the virtual machine settings

virsh edit <virtualhost>

Conclusions

In my opinion it seems like KVM is still a little bit immature or at least the tools to handle it. Maybe it would be a good idea to have one linux distro focusing on beeing the best platform for virtualization.

I am convinced that I can get even more bang for the bucks out of my installation if I learn how to tweak it even more, so if you have any hints, do not hesitate to contact me. Preferably as a comment to this blogpost.

Referenses:

• op5 Monitor, a nagios based enterprise monitor tool to monitor my environment
• check_libvirt, a nagios plugin done by op5 to monitor KVM
• check_diskio, a nagios plugin to monitor disk I/O
• check_cpu_stats, a nagios plugin to monitor cpu usage rather detailed
• Another blogpost which helped me
• KVM performance improvements and optimizations – Red Hat presentation

Before installing, read the release notes

Background

It is always a good idea to monitor the server hardware, in many cases the root cause of the probblem is hardware related like: a fan stops and the temperature gets to high, dust in the machine makes it to hot, disks that fails, memory corruption and so on.  This article will describe howto enable hardware monitoring on a HP Proliant running CentOS Linux and then howto collect the data with Nagios or op5 Monitor. The procedure is the same with RedHat Enterprise Linux and similiar with Suse Enterprise server.

The HP manuals and information is bloated with irrelevant information and I had to struggle several hours, ask collegues to get it running. I hope this blog article will help others to get monitoring of HP Proliant using HP Insight Manager easier to setup.

Installing the software on the target system

You need two packages from HP and can be downloaded from hp.com under "Support & Drivers". Search your hardware plattform and correct operating system:

• ProLiant Support Pack for Red Hat Enterprise Linux 5 (i686)  The latest Proliant Support Package 2010-03-02 name is psp-8.25.rhel5.i686.en.tar.gz
• HP System Health Application and Insight Management Agents for Red Hat Enterprise Linux 5 (x86). The latest name is hpasm-8.0.0-173.rhel5.i386.rpm

Install kernel source code and rpm tools:

# yum install kernel-devel rpm-build rpm-devel
 

Proliant Support Package is not supported on CentOS so you have to let the installer think it is a RedHat system. If you have a RHEL system skip the next steps.

# cp /etc/redhat-release /etc/redhat-release.backup

# echo "Red Hat Enterprise Linux Server release 5.4 (Tikanga)">/etc/redhat-release

Untar the Proliant Support Package

 # tar xzvf psp-8.25.rhel5.i686.en.tar.gz

Install the Proliant Support Package

# cd compaq/csp/linux/

# ./install825.sh

alot of text appears and some questions, answer them.

Install HP System Health Application and Insight Management Agents, and for some stupid reason it is in conflict with some other packages just installed. I solved it in a dirty way:

# rpm -i --force  --replacefiles --nodeps hpasm-8.0.0-173.rhel5.i386.rpm

Configure by running:

# hpa/etc/init.d/hpasm configure

and answer the questions.

Do not forget to restore /etc/redhat-release

# cp /etc/redhat-release.backup /etc/redhat-release

I did modify my /etc/snmp/snmp.conf to:

dlmod cmaX /usr/lib/libcmaX.so
rocommunity public
trapsink 10.1.1.20
syscontact peter@it-slav.net
syslocation PDC, Peters DataCenter

To test that you have installation and configuration work, run a snmpwalk from your Nagios or op5 Monitor host:

# snmpwalk -c public -v1  <ip-adress of your proliant box> 1.3.6.1.4.1.232

SNMPv2-SMI::enterprises.232.1.1.1.0 = INTEGER: 1
SNMPv2-SMI::enterprises.232.1.1.2.0 = INTEGER: 23
SNMPv2-SMI::enterprises.232.1.1.3.0 = INTEGER: 2
SNMPv2-SMI::enterprises.232.1.2.1.4.1.0 = INTEGER: 30
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.1.1 = INTEGER: 1
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.2.1 = STRING: "Compaq Standard Equipment Agent for Linux"
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.3.1 = ""
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.4.1 = Hex-STRING: 00 00 00 00 00 00 00
SNMPv2-SMI::enterprises.232.1.2.1.4.2.1.5.1 = STRING: "To gather Standard Equipment data for Linux."

...
 

Install check_hpasm on the Nagios or op5 Monitor host

The check_hpasm can be downloaded from Console Labs.

Unpack the tarball

# tar xzvf check_hpasm-4.1.2.tar.gz

Configure and compile

# ./configure --prefix=/opt/plugins/custom/hp-insight --with-nagios-user=monitor --with-nagios-group=users --enable-perfdata

...

# make

...

# make install

Test

# /opt/plugins/custom/hp-insight/libexec/check_hpasm -H  -C public

OK - System: 'proliant dl360 g3', S/N: '7J31LMW6N01D', ROM: 'P31 01/28/2004', hardware working fine, da: 1 logical drives, 1 physical drives | fan_1=50% fan_2=50% temp_1_cpu=16;50;50 temp_2_cpu=15;65;65 temp_3_ioBoard=21;56;56 temp_4_cpu=20;65;65

Congratulations your plugin and hw monitoring works!

Configure Nagios or op5 Monitor
 

checkcommands.cfg

# command 'check_hpasm'
define command{
    command_name                   check_hpasm
    command_line                   $USER1$/custom/libexec/check_hpasm -H $HOSTADDRESS$ -C $ARG1$
    }

services.cfg

# service 'Insight Manager'
define service{
    use                            default-service
    host_name                      humpa
    service_description            Insight Manager
    check_command                  check_hpasm!public
    contact_groups                 call_it-slav,it-slav_jabber,it-slav_mail
    }

Screenshoot, using ninja

 Useful links

• op5, a systems and network management company
• op5 Monitor, an enterpise monitor system based on Nagios
• Ninja, Nagios is now just awesome
• Nagios, enterprise monitoring based on opensource
• Hp support & drivers, a place to start looking for the HP software used in this article

But today I noticed that I’ve only got 1 mail so something must be wrong. I did send an email from my job to my private adress and got the email back with the following message:

<peter@it-slav.net>: cannot update mailbox /var/mail/peter for user peter.
    error writing message: File too large

After some troubleshooting I found the following in the postfix documentation:

mailbox_size_limit (default: 51200000)

The maximal size of any local(8) individual mailbox or maildir file, or zero (no limit). In fact, this limits the size of any file that is written to upon local delivery, including files written by external commands that are executed by the local(8) delivery agent.

This limit must not be smaller than the message size limit.

So I changed the mailbox_size_limit = 0 and now everything seems alright again.

1. Installation

Run command yum install net-snmp-utils

[root@dull etc]# yum install net-snmp-utils
Loading "fastestmirror" plugin
Loading "dellsysidplugin" plugin
...
...
Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 net-snmp-utils          i386       1:5.3.1-24.el5_2.2  updates           182 k
Installing for dependencies:
 net-snmp                i386       1:5.3.1-24.el5_2.2  updates           698 k

Transaction Summary
=============================================================================
Install      2 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 879 k
Is this ok [y/N]: y

Answer y

Downloading Packages:

(1/2): net-snmp-utils-5.3 100% |=========================| 182 kB    00:02
(2/2): net-snmp-5.3.1-24. 100% |=========================| 698 kB    00:06
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
 Installing: net-snmp                     ######################### [1/2]
  Installing: net-snmp-utils               ######################### [2/2]
Installed: net-snmp-utils.i386 1:5.3.1-24.el5_2.2
Dependency Installed: net-snmp.i386 1:5.3.1-24.el5_2.2
Complete!

Now it is installed

2. Configure

I’m careful so I do a backup of the snmpd config file.

[root@dull ~]# mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.org 

Create a new config file.

[root@dull ~]# vi /etc/snmp/snmpd.conf

rocommunity  public
syslocation  "PDC, Peters DataCenter"
syscontact  peter@it-slav.net

Start the snmpd service

[root@dull ~]# /etc/init.d/snmpd start

Do a snmpwalk to make sure it is working

[root@dull ~]# snmpwalk -v 1 -c public -O e 127.0.0.1
SNMPv2-MIB::sysDescr.0 = STRING: Linux dull 2.6.18-92.1.17.el5 #1 SMP Tue Nov 4 13:45:01 EST 2008 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (16748) 0:02:47.48
SNMPv2-MIB::sysContact.0 = STRING: peter@it-slav.net
SNMPv2-MIB::sysName.0 = STRING: dull
SNMPv2-MIB::sysLocation.0 = STRING: "PDC, Peters DataCentral"
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (1) 0:00:00.01
...

...

Yes, it is working

And finally, make sure snmpd starts next time you restart your machine.

[root@dull ~]# chkconfig snmpd on

3. Monitor example

Below is an example of how it looks using op5 Monitor a Nagios based Enterprise Monitor solution.

4. Useful links

• op5 Statistics, a cacti based graph tool
• op5 Monitor, an Enterprise Class Monitoring system based on Nagios
• Net-SNMP, an open source implementation of SNMP
• Cacti, an open source graph tool
• Nagios, the number 1 monitor tool

The script looks in /etc/fstab and compares it with the file systems mounted. If anything is missing, return CRITICAL and the name of the missed mountpoint. The script does not check that some system file systems are mounted i.e. /proc

I have uploaded this script to nagios exchange.

#!/bin/sh
#By peter@it-slav.net
#GPLv2

RESULT=0
TMPFILE=`mktemp /tmp/mount.XXXXXXXXXX`
FSTABMOUNTS=`grep -e '^#' -v /etc/fstab|grep -v  tmpfs |grep -v devpts|grep -v sysfs|grep -v proc|grep -v swap| awk '{print $2}'`
for i in $FSTABMOUNTS
do
        mountpoint $i > /dev/null
        if [ $? != "0" ]
        then
                echo -n "$i " >>$TMPFILE
                RESULT=2
        fi

done
#echo $RESULT
if [ $RESULT != "0" ]
then
        echo "is not mounted" >> $TMPFILE
        echo -n "CRITICAL: "
        cat $TMPFILE
else
        echo "OK: All disks mounted"
fi
rm $TMPFILE
exit $RESULT

Links:

• check_mounts.sh at Nagiosexchange
• op5 Monitor
• My other blog about the problems when I did a kernel upgrade

The first system is the machine that runs this blog. It runs Xen kernel with two virtual system.Luckily the webserver and database runs in Dom0. After update the system did not start, it seems like the Xen kernel is broken, so now I run non-Xen kernel.

The second system is my op5 Monitor system that monitor my environment. That changed the default kernel to Xen and that was broken, after chenged boot kernel to the non-Xen it works again.

The third machine is my fileserver, with a couple of disks. Two of the filesystems are XFS, and the new kernel did not have support for XFS so my large file systems could not be mounted. So I started the old kernel with XFS support.

This is not an acceptable behavior of an enteprise operating system.