Xen Server Configuration

If you plan to use Citrix's commercial XenServer, this article really doesn't apply. Their offering is based on CentOS (a community distribution of Red Hat), and is designed to be as appliance-like as possible. It also requires you to rely upon a Windows-based management tool, which is strange and may not be desirable. Depending on your needs, a more community-oriented distribution of the Xen hypervisor might hold more appeal. I've found that Debian has been an excellent platform to this end, thus this guide assumes you will be configuring your Xen environment on that distribution.

Since upgrading a Xen server to a newer distro entails a lot of changes which may affect guest operating systems running under it, I recommend the current stable build of Debian.¹⁾ However, this tutorial may apply in general terms to the community Xen stack shipped with many distributions of Linux, including Ubuntu Hardy and perhaps others. Whatever distro you settle on, using the 64-bit version is always preferable for performance and flexibility, assuming the hardware supports it. Note that a 64-bit Xen server will also support 32-bit guest OSes seamlessly.

• If many diverse guest domains are to share a single storage array, RAID-10 may be a better option than most other RAID types. It's fully redundant and its performance is very consistent in a variety of situations.
• Linux's 'md' RAID, when used on modern hardware, will quite often outperform even high-end hardware RAID controllers. If your budget is limited you might save some money by relinquishing that expensive RAID card, instead opting to add more RAM to share amongst your guest domains.
• Even if you choose to forgo hardware RAID, a battery backup unity for your storage controller is a good investment. That will allow you to safely utilize your drive cache without risking the integrity of your data in the event of a power failure.
• Keep in mind the I/O requirements of each guest domain your Xen server will be hosting. If a certain domain has particularly intense demands, e.g. a busy database server, it might be wise to isolate it on its own storage array. Otherwise, you could end up with an unacceptable amount of storage contention between domains.
• Plan ahead and estimate the space requirements of each guest domain to the best of your abilities. If you intend to use LVM, as recommended in this guide, you can re-size logical volumes later to give them more space. However, this usually means the volume will not be contiguous on its underlying storage, so it's always best to get it right the first time.

The host domain (dom0) shouldn't be running anything except for the guest domains (domU's), so unless you have special considerations, dom0's space requirements should be relatively minimal. I usually setup dom0's partitioning similarly to the following:

There are many ways to provide storage for domU's, but perhaps the most desirable is to give them an LVM logical volume. This is for flexibility and ease of archival. Create an LVM volume group on the remaining free space and activate it. You might name the volume group something like 'xenvg01', and each logical volume descriptively, e.g. 'domu01-root', 'domu01-swap', etc. If you already know how much space you want to allocate to guest domains you can create those logical volumes now, but don't bother mounting them anywhere just yet. Xen will be mounting the volumes directly as needed, and we don't want dom0 automatically mounting those partitions on boot.

After the OS installation and subsequent first boot, upgrade to current:

# aptitude update && aptitude dist-upgrade

Then install the xen-hypervisor-amd64 meta-package, Xen-enabled kernels, and a few other useful tools:

# aptitude install xen-hypervisor-amd64 linux-headers-2.6-xen-amd64 linux-image-2.6-xen-amd64 linux-modules-2.6-xen-amd64 openssh-server ntp vim htop saidar screen

Depending on which software you'll be utilizing, your guest domains might need optical media at some point. If so, you can use loopback devices to mount CD or DVD images which the domU's can utilize. Loopback devices can be very useful for virtualization, but they can cause overhead, so we shouldn't make this value arbitrarily high. A good amount is typically 0-2 loopback devices per domU, depending on your needs. For example, I can increase the default amount of device loopbacks from a max of 8 up to 16, and network loopbacks up to 32:

# vim /etc/modules

…and change or add the following lines like so:

loop max_loop=16
netloop nloopbacks=32

Next, we should change our environment variables to include safely-optimized cflags so that any modules we build take full advantage of the hardware. For example, on a 64-bit server, you'd append something like the following to /etc/environment:

CHOST="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="-march=native -O2 -pipe"

_{For more on safe cflags, check out this article over at the Gentoo wiki: http://en.gentoo-wiki.com/wiki/Safe_Cflags}

Afterwards you can install any necessary filesystem utilities, for example:

# aptitude install xfsprogs xfsdump ntfs-3g ntfsprogs

Now edit the /etc/xen/xend-config.sxp file to make sure that dom0 always has a sane amount of memory available to it. For example:

(dom0-min-mem 1024)

One gigabyte may seem like a lot for dom0, since it shouldn't be running any unnecessary services. However, keep in mind that dom0 essentially runs on bare metal, and as such it will manage all device IO for any HVM domains. For this reason 1GB is a reasonable amount. If your memory is especially constrained you can use less, but I would not recommend any less than 512MB.

You should also change the network-script entry to make it use the network-bridged script rather than the network-dummy default value:

(network-script network-bridge)

You'll need to edit the bootloader, e.g. /etc/grub/menu.lst in most cases, and append the dom0_mem option to xenhopt using the same value you set above for 'dom0-min-mem' earlier:

# vim /boot/grub/menu.lst
...
         # xenhopt=dom0_mem=1024M

…Then update the bootloader:

# update-grub

Next, we can increase the kernel network buffers and make the OS avoid touching swap (swapping on shared drives is especially bad):

# vim /etc/sysctl.conf

...

          # Add lots more TCP buffer space for high-latency connections
          net.core.wmem_max = 12582912
          net.core.rmem_max = 12582912
          net.ipv4.tcp_rmem = 10240 87380 12582912
          net.ipv4.tcp_wmem = 10240 87380 12582912

          # An option to enlarge the transfer window
          net.ipv4.tcp_window_scaling = 1

          # Enable timestamps as defined in RFC1323
          net.ipv4.tcp_timestamps = 1

          # Enable select acknowledgments
          net.ipv4.tcp_sack = 1

          # If set, TCP will not cache metrics on closing connections
          net.ipv4.tcp_no_metrics_save = 1

          # Set maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them
          net.core.netdev_max_backlog = 5000
          
          # Force the kernel not to swap idle vm space unless all RAM is expended. Default is '60'.
          vm.swappiness = 0

...

# sysctl -p

At this point, you should reboot the Xen server so that it can load the appropriate Xen-enabled kernel, subsequently activating your environmental changes from earlier.

When the OS has come back up, verify it's running the new Xen kernel:

$ uname -a
Linux ns041a 2.6.26-2-xen-amd64 #1 SMP Tue Aug 31 11:17:26 UTC 2010 x86_64 GNU/Linux

Once that's established, we should purge out all the other installed kernels. This is to conserve space on dom0, but most importantly to avoid accidentally installing an upgraded kernel in the future that doesn't contain Xen support. The process would be something like the following:

$ dpkg -l | grep linux

… which will give us a list of all the installed kernels (among other things). Then we purge all the unnecessary ones, e.g.:

# aptitude purge linux-image-2.6-amd64

There are two methods to speed up qemu support in Linux: KVM (Kernel-based Virtual Machine) or kqemu (qemu kernel module). KVM offers better performance, but relies on having a kernel >=2.6.20 and hardware virtualization features only available in new model CPUs. Therefore, if the necessary prerequisites can be met, use KVM. If not, use kqemu. ONLY INSTALL ONE OR THE OTHER; by nature kqemu and KVM cannot co-exist successfully on the same system.

You can check if your CPU supports KVM like so:

$ egrep '(vmx|svm)' --color=always /proc/cpuinfo

…if the above returns nothing, your CPU is unsupported in KVM. In that case, use kqemu.

_{Note: kqemu support is reportedly being phased out, so KVM may be the only option for qemu accelleration in the future.}

If you're in an environment that supports KVM, installation is easy:

# aptitude install kvm

Once installed, you will need to load the approproate module for your CPU in order to reap its benefits. For an Intel CPU, the module is called kvm-intel, and likewise for AMD it's kvm-amd.

For example, if we have a Xeon CPU:

# modprobe kvm-intel

Assuming the module loads without error, you can go ahead and have the kvm module load by default with the system:

# echo kvm-intel >> /etc/modules

Install the kqemu module, which provides a kernel module that helps accelerate qemu, along with the headers for the running kernel:

# aptitude install qemu kqemu-common kqemu-source module-assistant linux-headers-`uname -r`

Next, build the kqemu module with module-assistant:

# m-a a-i kqemu-source

If all goes as planned, load the module:

# modprobe kqemu

Assuming the module loads without error, you can go ahead and have kqemu load by default with the system:

# echo kqemu >> /etc/modules

Xen provides very secure segmentation between the hypervisor and each guest domain. But the dom0 OS is used to manage all other child domains, and therefore has direct control over them. Since compromising dom0 is equivalent to getting local access to each and every domU, some extra precautions are needed. Here are a few suggestions:

• Have dom0 run on a dedicated interface which is not shared with any domU. Place this interface on a management network which is isolated from your production network.
• If dom0 cannot be isolated, limit attack vetors by setting local services to listen on loopback only.
• In case you still need further hardening, consider installing a firewall script to protect dom0's network stack.

Unless you will be exporting an NFS share from your dom0, you should configure services like the portmapper (RPC) to listen only on localhost. Assuming the OS used is Debian:

# dpkg-reconfigure portmap

Other services, like Posftfix, should be limited in the same manner:

# vim /etc/postfix/master.cf

...

          127.0.0.1:smtp      inet  n       -       -       -       -       smtpd

...

# postfix reload

And similarly with NTP:

# vim /etc/ntp.conf
...
          restrict 127.0.0.1
...

You should do likewise for any other daemons which will be running on dom0.

Depending on how and where the server is to be deployed you might also consider applying a simple firewall script, perhaps dtc-xen-firewall:

# aptitude install dtc-xen-firewall

I will leave the details of its configuration as an exercise for the reader.

Now you can create logical volumes as desired for your domU's. In this example, I'll create a very simple guest domain with only a 200GB root and a 2GB swap volume. To create a 200GB logical volume under LVM volume 'xenvg01' called 'domu01-root':

# lvcreate -L200000 -ndomu01-root xenvg01

_{For more info on LVM, see the official HowTo: http://www.tldp.org/HOWTO/LVM-HOWTO/index.html}

Note that on Linux or guests, you should format the partition(s) as desired. For Windows or other guests that don't support a simple bootstrap install, you should leave the partition(s) unformatted. In that case, the guest OS will handle the formatting itself as part of its installation.

Since we are installing this guest via network bootstrap, we will go ahead and format each volume with our filesystem of choice, e.g.:

# mkfs.ext3 /dev/xenvg01/domu01-root

…and for the swap volume:

# mkswap /dev/xenvg01/domu01-swap

Afterwards, create a directory as a mount point for domU volumes. In this example I will use /xen for this purpose:

# mkdir /xen

Under this, you can create a mount point for each domU guest as needed, e.g. /xen/domu01, /xen/domu02, etc.

# mkdir /xen/domu01

When ready, mount the volume in question:

# mount /dev/xenvg01/domu01-root /xen/domu01

To install a 64-bit build of Lenny as 'domu01', we can employ the ever useful debootstrap tool:

# debootstrap --arch amd64 lenny /xen/domu01 http://ftp.debian.org/debian

Once finished, prepare a chroot environment for /xen/domu01 so we can configure our guest domain:

# mount --bind /dev /xen/domu01/dev
# mount proc /xen/domu01/proc -t proc

Now is a good time to batch edit a few config files on the new bootstrap installation to suit the situation, namely /etc/resolv.conf, /etc/hosts, /etc/hostname, and /etc/network/interfaces. I will assume you understand what the contents of each file should look like in your own environment:

# vim /xen/domu01/etc/resolv.conf /xen/domu01/etc/hosts /xen/domu01/etc/hostname /xen/domu01/etc/network/interfaces

After that's complete, we can drop into the chroot to finish up the installation:

# chroot /xen/domu01 /bin/bash

From within the domU's chroot environment, we should set our locale:

# aptitude install locales && dpkg-reconfigure locales

Then create a bare-bones /etc/fstab as follows, making changes if/where appropriate (e.g. FS type):

/dev/sda1               /               ext3    noatime        1       2
/dev/sda2               none            swap    sw             0       0

Upgrade to current here as well:

# aptitude update && aptitude dist-upgrade

…then install the appropriate kernel modules. In Debian, you can simple do the following (assuming the Xen-enabled kernel is already running; otherwise choose the appropriate module package):

# aptitude install linux-modules-`uname -r`

Next, install handy tools:

# aptitude install openssh-server bc vim sudo udev htop screen

…and XFS utilities if the domU is running or will be interacting with that filesystem:

# aptitude install xfsprogs xfsdump

Swapping on shared storage is especially undesirable; lessen the aptitude of the kernel to swap idle vm space (default value is '60'):

# echo "vm.swappiness = 0" >> /etc/sysctl.conf

Before logging out, you should take the opportunity to either set a password for the built-in root account:

# passwd

…or if you wanted a more Ubuntu-like environment, you can of course disable root and create a local admin account instead.

Once finished, we're done with the chroot environment so we can exit:

# exit

If you're done editing configuration files in your new host, unmount all of the partitions we mounted earlier:

# umount -l /xen/domu01/dev /xen/domu01/proc /xen/domu01

Now create the first guest OS's Xen configuration file named domu01-<description>.cfg in the /etc/xen/ folder, where <description> might be something like the hostname of the new domain.

Here's my boilerplate Linux domu.cfg to get you started:

• Xen Linux client config boilerplate

After you're satisfied with the state of the configuration file, you can start the first domU like this:

# xm create /etc/xen/domu01.cfg -c

Note that the '-c' will start the domU in console mode, which is the same as being connected to a console via serial port on the domU. This is helpful for troubleshooting, but once you're sure of its stability, you can leave that flag off to start the domU in the background.

Log in using the account you configured earlier and verify the host's functionality. If everything looks as expected, your first guest OS (domU01) is ready now. To exit the console interface, use 'Ctrl + ]'

Some operating systems can't be installed with the simple bootstrap procedure described above. As a result, they must be installed from a CD-ROM, ISO image, or network resource. In the case of Windows and other operating systems which cannot be para-virtualized in Xen, they must run in an HVM environment.

First we must prepare a partition onto which our fully-virtualized OS, e.g. Windows, will be installed. For instance, if we wanted a 40GB logical volume called “domu02” in the volume group “xenvg01”, we would do as follows:

# lvcreate -L40000 -ndomu02 xenvg01

_{For more info on LVM, see the official HowTo: http://www.tldp.org/HOWTO/LVM-HOWTO/index.html}

With an HVM guest like Windows, we must create the guest OS's Xen configuration file in the /etc/xen/ folder before we can proceed with the installation. Here's my boilerplate Windows domu.cfg for reference:

• Xen Windows client config boilerplate

For the fastest and most convenient CD or DVD-based install, it's prudent to make an ISO image of the install disc and mount that.

I prefer to make a directory for resources such as installation discs. For example:

# mkdir /xen/resources

_{If you will be housing many images in this directory, be sure that it has adequate capacity. On a stand-alone Xen server, perhaps store the images on their own partition. In an environment with many Xen servers, it might be better to have a single ISO repository exported R/O globally via NFS.}

For most discs, the 'dd' utility does an adequate job of creating a working image file. Assuming dom0's optical drive is designated as /dev/sr0 and we're dumping a 32-bit Windows XP install CD, we'd do something like the following:

# dd if=/dev/sr0 of=/xen/resources/windows_xp_pro_32.iso

In the case of mounting a CD Image for installation, creating a loopback device seems to be the most consistent and reliable way. To do this, we'll use the excellent 'losetup' tool. Assuming we wanted to mount the image at /xen/resources/windows_xp_pro_32.iso, we could do as follows:

# losetup -f /xen/resources/windows_xp_pro_32.iso

Note that the '-f' argument will use the first available loopback device. To list all of the used loopback devices, you can do the following:

losetup -a

…and the newly mounted loopback device should be among those listed.

To free up a loopback device when you're done with it:

losetup -d /dev/loop0

Now that we have the installation medium ready, we need to get the domU setup with a virtual ROM device to load said media. This can be accomplished with a simple change to the existing domU config file to add a new device. In this example, I already have a line as follows in my /etc/xen/domu02.cfg file:

disk        = [ 'phy:/dev/xenvg01/domu02,hda,w' ]

Since in my case I have already mounted an install disc image at /dev/loop0, I will change the config file to add a virtual CD-ROM device as follows:

disk        = [ 'phy:/dev/xenvg01/domu02,hda,w', 'phy:/dev/loop0,hdc:cdrom,r' ]

…where hdc indicates the virtual device path to be presented to the guest OS, and r implies a read-only status.

Next, we will need to set the new virtual ROM device to be the bootable device on the system. We will need to find the boot line:

boot        = 'c'

…which designates that the device a DOS BIOS would consider as “drive C”, or the first IDE, device, is to be considered the first boot device. We will need to change it so that the second IDE device, i.e. “drive D” will be the primary boot device:

boot        = 'd'

Once the Guest OS environment is prepped and the install media is ready, we can proceed with the actual installation process. You can connect to the Xen server's running VNC interface to simulate sitting in front of a dedicated machine. That said, the installation process itself is exactly as one would normally install Windows.

Once the installation is complete, you will also want to install the open-source Xen drivers for Windows (GPLPV), which will boost IO performance substantially. Instructions on this process can be found here:

http://www.jolokianetworks.com/70Knowledge/Virtualization/Open_Source_Windows_Paravirtualization_Drivers_for_Xen

And the GPLPV drivers themselves may be found here: http://www.meadowcourt.org/downloads/

Note that MAC addresses will change once the drivers have been installed, and previous chipset or network devices will show as missing in the device manager. You should simply disable these rather than removing them in case you need to roll back to using the compatibility layer in the future.

After the GPLPV drivers have been installed and the system rebooted, activate RDP as you normally would for remote administration. Confirm that it works and remote connections can be established reliably, then disable the VNC viewer for the domU in question and remove the CDROM loopback entry. The domU will need to be destroyed and restarted before the changes go into effect.

Most Unix-like OS'es can be run as a Xen guest with para-virtualization for best performance. Unfortunately, many OSes are difficult to install via the convenient bootstrap method mentioned above. That being the case, we simply need to add a CD-ROM device or mount an ISO image as a loopback device, then install from that. You can also install from a network resource, though that's out of the scope of this tutorial.

First we must prepare a partition onto which our guest OS, e.g. FreeBSD, will be installed. For instance, if we wanted a 40GB logical volume called “domu03” in the volume group “xenvg01”, we would do as follows:

# lvcreate -L40000 -ndomu03 xenvg01

_{For more info on LVM, see the official HowTo: http://www.tldp.org/HOWTO/LVM-HOWTO/index.html}

For an illustration of a Xen config for an OS that will not boot using the system's kernel, here's my boilerplate domu.cfg for FreeBSD:

• Xen *nix client config boilerplate

Should you want more information on configuring Xen for FreeBSD specifically, see the FreeBSD docs on virtualization: http://www.freebsd.org/doc/en/books/handbook/virtualization-guest.html

Similarly, in the case of other non-bootstrapped POSIX OS'es you may need to consult their respective docs for distro-specific information.

I prefer to make a directory for resources such as installation discs:

# mkdir /xen/resources

_{If you will be housing many images in this directory, be sure that it has adequate capacity. On a stand-alone Xen server, perhaps store the images on their own partition. In an environment with many Xen servers, it might be better to have a single ISO repository exported R/O globally via NFS.}

Having done that, it's prudent to simply download the CD image directly from the vendor to the resources directory. For instance,

# wget ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/7.1/7.1-RELEASE-amd64-dvd1.iso -O /xen/resources/freebsd_x86_64_v7_1.iso

Alternately, if you've already got the physical install media, you can make an ISO image of the install disc and mount that.

For most discs, the 'dd' utility does an adequate job of creating a working CD image. Assuming dom0's optical drive is designated as /dev/sr0 and we're dumping a 64-bit FreeBSD 7.1 disc, we'd do something like the following:

# dd if=/dev/sr0 of=/xen/resources/freebsd_x86_64_v7_1.iso

In the case of mounting a CD Image for installation, creating a loopback device seems to be the most consistent and reliable way. To do this, we'll use the excellent 'losetup' tool. Assuming we wanted to mount the image at /xen/resources/freebsd_x86_64_v7_1.iso, we could do as follows:

# losetup -f /xen/resources/freebsd_x86_64_v7_1.iso

Note that the '-f' argument will use the first available loopback device. To list all of the used loopback devices, you can do the following:

losetup -a

…and the newly mounted loopback device should be among those listed.

Now that we have the installation medium ready, we need to get the domU setup with a virtual ROM device to load said media. This can be accomplished with a simple change to the existing domU config file to add a new device. In this example, I already have a line as follows in my /etc/xen/domu2.cfg file:

disk        = [ 'phy:/dev/xenvg01/domu03,hda,w' ]

Since in my case I have already mounted an install disc image at /dev/loop0, I will change the config file to add a virtual CD-ROM device as follows:

disk        = [ 'phy:/dev/xenvg01/domu03,hda,w', 'phy:/dev/loop0,hdc:cdrom,r' ]

…where hdc indicates the virtual device path to be presented to the guest OS, and r implies a read-only status.

Next, we will need to set the new virtual ROM device to be the bootable device on the system. We will need to find the boot line:

boot        = 'c'

…which designates that the device a DOS BIOS would consider as “drive C”, or the first IDE, device, is to be considered the first boot device. We will need to change it so that the second IDE device, i.e. “drive D” will be the primary boot device:

boot        = 'd'

After you're satisfied with the state of the configuration file, you can start the first domU like this:

# xm create /etc/xen/domu03.cfg -c

Note that the '-c' will start the domU in console mode, which is like being connected to that machine via serial console.

Now proceed with the actual installation process. This will not differ greatly from how you would normally install the guest OS in question on dedicated hardware.

To start a domU normally, do the following:

# xm create /etc/xen/domu<#>.cfg

…where <#> is the number of the domU in question. Alternately, to start it with a connected console for troubleshooting:

# xm create /etc/xen/domu<#>.cfg -c

To disconnect from a running console, press:

'Ctrl + ]'

To reconnect again, you would run:

# xm console <DOMU-NAME>

Where DOMU-NAME is the actual name of the domU to which you wish to connect. In dom0 (i.e. the Xen host server), you can list running domU's like so:

# xm list

…or profile their resource load with the 'xentop' command:

# xentop

For general information including free memory for allocation:

# xm info

If you need more information on the xm tool:

# xm help

You can “hot add” volumes to your guest domains by using the xm block-attach command. According to the xm man page, the syntax is as follows:

xm block-attach <Domain> <BackDev> <FrontDev> <Mode> [BackDomain]

Since the above is a bit confusing, here's a real world example. Supposing I want to hot-plug a physical HDD partition on dom0 (/dev/sdb2) to a guest domain called “web01” as a writable volume at /dev/hdb2, I would do the following:

xm block-attach web01 phy:/dev/sdb2 /dev/hdb2 w

The drive should be available for use in the domU immediately. You can confirm this by running the 'dmesg' command on the guest domain. It should contain an entry about the hotplug event, for example:

[1166306.889163] blkfront: hdb2: barriers enabled

If you'd like to make the change permanent, don't forget to append your changes to the 'disk' subsection of your domU's Xen client configuration.

If desired, you can bind domU's to a specific interface – be it a physical interface, VLAN, bonded link, etc. First, you will need to bridge the interfaces in question. In this example, I will be adding bridges to eth0 and eth1, which are on two different subnets, so that domU's can be bound to either. Xen provides some scripts to create domU-friendly bridges on existing interfaces, so we'll just edit /etc/network/interfaces to call them:

# The loopback network interface
auto lo
iface lo inet loopback

# Physically connected to DMZ3
auto eth0
iface eth0 inet static
        address 10.20.3.5
        netmask 255.255.255.0
        gateway 10.20.3.1

# Bridge eth0 for Xen
auto peth0
iface peth0 inet manual
        up /etc/xen/scripts/network-bridge start netdev=eth0
        down /etc/xen/scripts/network-bridge stop netdev=eth0

# Physically connected to DMZ1
auto eth1
iface eth1 inet static
        address 10.20.1.5
        netmask 255.255.255.0
        gateway 10.20.1.1

# Bridge eth1 for Xen
auto peth1
iface peth1 inet manual
        up /etc/xen/scripts/network-bridge start netdev=eth1
        down /etc/xen/scripts/network-bridge stop netdev=eth1

Afterwards, all that's left is to define which interface(s) each domU will be bound to. For example, if I wanted a guest OS to be bound specifically to eth0, I would append bridge=eth0 to the vif section of its conf file.

To have domUs start automatically with the Xen daemon, just symlink the configs you want to autolaunch to '/etc/xen/auto/'. If /etc/xen/auto doesn't exist, create it first then create your symlink, e.g.:

# mkdir /etc/xen/auto/ && ln -s /etc/xen/domu01.cfg /etc/xen/auto/

The domains will start in alpha-numeric order. If you wish your domUs to startup in a more specific order, an easy way to achieve this is by prefixing each symlink with numbers. For example:

# ln -s /etc/xen/domu02.cfg /etc/xen/auto/00-domu02.cfg
# ln -s /etc/xen/domu03.cfg /etc/xen/auto/01-domu03.cfg
# ln -s /etc/xen/domu01.cfg /etc/xen/auto/02-domu01.cfg

In some situations, especially lab environments, it might be very valuable to have multiple live snapshots of your domUs. If you've taken my advice and put domU's on their own LVM logical volumes, you can use this facility to create snapshots. LVM2 inparticular includes the capability for creating R/W snapshots, regardless of which filesystem the volume is formatted with, so the path of least resistance may be to run your domU's from a snapshot. Or, if you will be creating many snapshots along the way, it may be easier for you to run the domU's off of the original LVM and simply create backup snapshots as needed. In either case, keep in mind that “dragging” LVM snapshots incurs a performance penalty, so this feature should not be used arbitrarily.

In this example, I'll be using the former approach, running the domU's off of the snapshot and leaving the original LVM partition intact. This way, I can always go back to my earlier pristine OS image if I have a problem with a domU or just need a fresh start.

First, I might shutdown my pristine domU gracefully to ensure a 100% consistent state. Next, I create a snapshot of my pristine VG (/dev/xenvg01/domu02) and name it something appropriate (domu02.live):

lvcreate --snapshot --size=60000 --name=domu02.live /dev/xenvg01/domu02

In the example above I've used an arbitrary snapshot size of 60GB, but this is something you should consider carefully before implementing. An LVM snapshot works as a delta, so it will only grow in terms of changes made from the original VG. This means it can potentially be much smaller than the original VG. On the other hand, the snapshot will stop working if the delta exceeds the space you've allotted for it. Thus, if the domU changes little, you can get away with using only a small fraction of the original VG; perhaps 10-25%. In a case where you need to be absolutely certain the domU will never overrun your snapshot, the safe choice is to create a snapshot that's exactly the size of the original VG.

The next change that must be made if you are running your domU's from a snapshot is to change the config file to reflect this. For example,

disk        = [ 'phy:/dev/xenvg01/domu02,hda,w']

becomes:

disk        = [ 'phy:/dev/xenvg01/domu02.live,hda,w']

…after which you can start your domU as normal.

Continuing the example above, let us assume that our pristine image is housed at /dev/xenvg01/domu02, and that our live version is in an undesirable state. We will simply take down the domU in question (e.g. domu02-broken_windows):

xm destroy domu02-broken_windows

We can then remove the unwanted LVM snapshot:

lvremove -f /dev/xenvg01/domu02.live

…and create a new one from the pristine source:

lvcreate --snapshot --size=60000 --name=domu02.live /dev/xenvg01/domu02

When this is finished, we simply start the domU as usual. It should come up in its pristine state as if nothing had ever gone wrong.

It's nice to be able to see trends on how each domain is consuming resources over time. Luckily, Xen's data is relatively easy to graph with RRD. First you'll need the RRD tool and it's prime dependency, python:

# aptitude install rrdtool python

Next, copy the contents of the data and graph generator scripts linked below to /usr/local/sbin/ :

• Xen Domain-0 RRD data generator
• Xen Domain-0 RRD graph generator

…then make them executable:

# chmod 755 /usr/local/sbin/xen*.py

You'll also need a place for the RRD data to go:

# mkdir -p /var/www/xen-rrd/data/

Finally, add entries to the system-wide crontab – e.g. /etc/crontab.d/xen-rrd – in order to gather data and create graphs automatically:

* * 	* * * 	root	nice -n 19 /usr/local/sbin/xenupdate.py /var/www/xen-rrd/data/
*/5 * 	* * *	root	nice -n 19 /usr/local/sbin/xengraph.py /var/www/xen-rrd/data/

Xen uses the 'xen' clocksource by default, which is shared between dom0 and all of the domU's. In theory this is efficient and desirable. Unfortunately, on some hardware this is nowhere near as stable as the normal Linux kernel timer and may cause problems. In my experience this has been especially true on Intel hardware, but not so on AMD – your mileage may vary. If you are seeing 'clocksource/0: Time went backwards' in your kernel logs, I recommend decoupling the domU's clock from dom0 and instead setting jiffies as the default clocksource.

# echo "xen.independent_wallclock = 1" >> /etc/sysctl.conf
# sysctl -p
# echo "jiffies"> /sys/devices/system/clocksource/clocksource0/current_clocksource

You'll also need to edit the bootloader, e.g. /etc/grub/menu.lst in most cases, and set append the clocksource=jiffies option to defoptions, xenhopt and xenkopt. For instance:

# vim /boot/grub/menu.lst
...
         # defoptions=clocksource=jiffies
         # xenhopt=clocksource=jiffies dom0_mem=1024M
         # xenkopt=console=tty0 clocksource=jiffies
...

…Then update the bootloader:

# update-grub

Now edit each domU's config file, appending clocksource=jiffies to its boot options; e.g:

# vim /etc/xen/domu01.conf

...

extra = 'xencons=tty1 clocksource=jiffies'

# echo "xen.independent_wallclock = 1" >> /etc/sysctl.conf
# sysctl -p
# echo "jiffies"> /sys/devices/system/clocksource/clocksource0/current_clocksource

And don't forget to install and configure an NTP daemon to keep the time synced:

# aptitude install ntp