As an it slave, I have the same problems as most datacenters:
- Running out of space because of more and more machines get into the datacenter
Added to the "normal" datacenter problems, I also have to deal with:
- Wife acceptance
- Noice, my neighbours has complained about the noice outside my basement.
To address this issues I will run a consolidation project to get fewer machines by using virtualization. As virtualization engine, KVM is choosed.
This article will describe some pitfalls I run into and how I solved them.
KVM seems to be the future for virtualization within the Linux area since Xen has done some mistakes.
As I have good experience of running CentOS and has KVM included I decided to use CentOS as the plattform for my virtualization.
To let the virtual boxes has full access to the network you need create a network bridge, it is rather straightforward and well documented so I will skip this part. I used the documentation on Red Hat customer support. Why KVM requires a bridge instead of a normal NIC is another question
One handy feature with virtualization is that the virtual systems can be one big fat file at the physical host, this gives the possibility to make full backups without turning the system down a.k.a snapshots.
However to get snapshot to work the filetype must be qcow2 and raw is the default format.
It is possible to convert from raw to qcow2 format by running the command:
qemu-img convert -f raw -O qcow2 <virtualhost>.img <virtualhost>.img.qcow2
Remember to turn of your virtual system by running:
virsh shutdown <virtualhost>
After the conversion:
virsh edit <virtualhost>
modify the type and path, i.e.
<driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/web.img.qcow2'/>
Start your virtual system:
virsh start <virtualhost>
Take a snapshot with:
qemu-img snapshot -c <snapshotname> <virtualhost>.img.preallc.qcow2
To create a full image of your snapshot you need to compile a more resent version of qemu-img then shipped with CentOS 6.2, I download a later version and compiled it from here, i renamed the new to qemu.img2 and copied it to /usr/local/bin
qemu-img2 convert -p -f qcow2 -O qcow2 -s <snapshotname> <virtualhost>.img.preallc.qcow2 <targetpath>
Beware: This command can use all your resources and may affect all your virtual systems, use nice and ionice to prevent it.
After installing just a few virtual system I noticed that my physical host got alot of I/O wait, it can be seen by using tools like top.
To monitor this I installed the nagios plugins:
- check disk io, can be found here
- check cpu stats, can be found here
- check_libvirt, developed by op5 and an Howto
However it seems like that though my virtual systems did more or less nothing they caused alot of disk I/O on the physical system.
I did some investigations and read quite many articles, fiddled and tested. The following is what worked for me in my setup.
#1 Change disk scheduler
The disk scheduler can be changed on runtime by modify the file:
to see what scheduler you use now:
Change sda to the device you have.
The scheduler that worked best for me is deadline
echo deadline > /sys/block/sda/queue/scheduler
#2 Mount with noatime
A feature in Unix and other Unixlike systems like Linux is that it normaly stores when a file is accessed. So one read always produce a write and if you are using raid like morroring this get worse, one reade always generate several writes. This creates alot of overhead for a feature soldom used.
So change /etc/fstab so it will mount the file systems with noatime.
UUID=a290aa4b-635c-45fa-b144-1fbef90b3735 / ext4 defaults,noatime 1 1
#3 Preallocation disk images
A real boost that is hidden in the featureset and not shown in the virtualmachine gui is using preallocation disk images. They cannot be created from the GUI so I install the virtual machine, turn it of and convert it afterwards.
qemu-img convert -f qcow2 -O qcow2 -o preallocation=metadata <virtualhost>.img.qcow2 <virtualhost>.img.preallc.qcow2
Change the path to new image name by editing the virtual machine settings
virsh edit <virtualhost>
In my opinion it seems like KVM is still a little bit immature or at least the tools to handle it. Maybe it would be a good idea to have one linux distro focusing on beeing the best platform for virtualization.
I am convinced that I can get even more bang for the bucks out of my installation if I learn how to tweak it even more, so if you have any hints, do not hesitate to contact me. Preferably as a comment to this blogpost.
- op5 Monitor, a nagios based enterprise monitor tool to monitor my environment
- check_libvirt, a nagios plugin done by op5 to monitor KVM
- check_diskio, a nagios plugin to monitor disk I/O
- check_cpu_stats, a nagios plugin to monitor cpu usage rather detailed
- Another blogpost which helped me
- KVM performance improvements and optimizations – Red Hat presentation