Welcome to It-Slav.Net blog
Peter Andersson
peter@it-slav.net

I've already got a female to worry about. Her name is the Enterprise.
-- Kirk, "The Corbomite Maneuver", stardate 1514.0

Background

As an it slave, I have the same problems as most datacenters:

  • Running out of space because of more and more machines get into the datacenter
  • Overheating
  • Powerconsumption

Added to the "normal" datacenter problems, I also have to deal with:

  • Wife acceptance
  • Noice, my neighbours has complained about the noice outside my basement.

To address this issues I will run a consolidation project to get fewer machines by using virtualization. As virtualization engine, KVM is choosed.

This article will describe some pitfalls I run into and how I solved them.

 

 

KVM plattform

KVM seems to be the future for virtualization within the Linux area since Xen has done some mistakes.

As I have good experience of running CentOS and has KVM included I decided to use CentOS as the plattform for my virtualization.

 

Issues

Bridge network

To let the virtual boxes has full access to the network you need create a network bridge, it is rather straightforward and well documented so I will skip this part. I used the documentation on Red Hat customer support. Why KVM requires a bridge instead of a normal NIC is another question :-)

 

Snapshots

One handy feature with virtualization is that the virtual systems can be one big fat file at the physical host, this gives the possibility to make full backups without turning the system down a.k.a snapshots.

However to get snapshot to work the filetype must be qcow2 and raw is the default format.

It is possible to convert from raw to qcow2 format by running the command:

qemu-img convert -f raw -O qcow2 <virtualhost>.img <virtualhost>.img.qcow2

Remember to turn of your virtual system by running:

virsh shutdown <virtualhost>

After the conversion:

virsh edit <virtualhost>

modify the type and path, i.e.

      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/web.img.qcow2'/>

Start your virtual system:

virsh start <virtualhost>

 

Take a snapshot with:

qemu-img snapshot -c <snapshotname> <virtualhost>.img.preallc.qcow2

 

To create a full image of your snapshot you need to compile a more resent version of qemu-img then shipped with CentOS 6.2, I download a later version and compiled it from here, i renamed the new to qemu.img2 and copied it to /usr/local/bin

qemu-img2 convert -p -f qcow2 -O qcow2 -s <snapshotname> <virtualhost>.img.preallc.qcow2 <targetpath>

Beware: This command can use all your resources and may affect all your virtual systems, use nice and ionice to prevent it.

 

Disk I/O

After installing just a few virtual system I noticed that my physical host got alot of I/O wait, it can be seen by using tools like top.

To monitor this I installed the nagios plugins:

  • check disk io, can be found here
  • check cpu stats, can be found here
  • check_libvirt, developed by op5 and an Howto

However it seems like that though my virtual systems did more or less nothing they caused alot of disk I/O on the physical system.

I did some investigations and read quite many articles, fiddled and tested. The following is what worked for me in my setup.

#1 Change disk scheduler

The disk scheduler can be changed on runtime by modify the file:

/sys/block/sda/queue/scheduler

to see what scheduler you use now:

cat /sys/block/sda/queue/scheduler

Change sda to the device you have.

The scheduler that worked best for me is deadline

echo deadline > /sys/block/sda/queue/scheduler

 

#2 Mount with noatime

A feature in Unix and other Unixlike systems like Linux is that it normaly stores when a file is accessed. So one read always produce a write and if you are using raid like morroring this get worse, one reade always generate several writes. This creates alot of overhead for a feature soldom used.

So change /etc/fstab so it will mount the file systems with noatime.
An axample:

UUID=a290aa4b-635c-45fa-b144-1fbef90b3735 /                       ext4    defaults,noatime        1 1

 

#3 Preallocation disk images

A real boost that is hidden in the featureset and not shown in the virtualmachine gui is using preallocation disk images. They cannot be created from the GUI so I install the virtual machine, turn it of and convert it afterwards.

qemu-img convert -f qcow2 -O qcow2 -o preallocation=metadata <virtualhost>.img.qcow2 <virtualhost>.img.preallc.qcow2

Change the path to new image name by editing the virtual machine settings

virsh edit <virtualhost>

 

Conclusions

In my opinion it seems like KVM is still a little bit immature or at least the tools to handle it. Maybe it would be a good idea to have one linux distro focusing on beeing the best platform for virtualization.

I am convinced that I can get even more bang for the bucks out of my installation if I learn how to tweak it even more, so if you have any hints, do not hesitate to contact me. Preferably as a comment to this blogpost.

 

Referenses:

Share

2 Responses to “KVM virtualization, some best practice, part 1”

  1. Joe Libero Says:

    Hi!
    According https://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_hellwig.pdf page 26 you should consider using cache=writethrough instead of cache=none. See page 22-25 as well.
    Thanks for sharing!

  2. peter Says:

    Thanks for the hint.

    Unfortunatly my biggest problem is disk I/O so now I run my disk images nfs mounted from a FreeNAS system or iSCSI. Using cache=writhethrough is even worse when it comes to performance.

    I have not figured out why the local disk is slower then nfs mounted.

Leave a Reply


× eight = 40





Book reviews
FreePBX 2.5
Powerful Telephony Solutions






Asterisk 1.6
Build a feature rich telephony system with Asterisk






Learning NAGIOS 3.0





Cacti 0.8 Network Monitoring,
Monitor your network with ease!