In this post I will demonstrate how I am setting up rather big infrastructures (> 10 DomUs, >2 Dom0s)

Networking

First, I create a bridged network interface for my DomUs. In this case, it is a Dom0 in a private LAN.

If this is not yet installed, install the bridging utilities:

# sudo aptitude install bridge-utils

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Local
auto lo
iface lo inet loopback

# LAN interface
auto eth0
iface eth0 inet manual
        post-up ifconfig eth0 0.0.0.0 up
        pre-down ifconfig eth0 0.0.0.0 down


# Bridge to LAN 
auto xenbr0
iface xenbr0 inet static
  address 10.0.0.1
  netmask 255.255.255.0
  gateway 10.0.0.254
  dns-nameservers 10.0.0.254

  # configure the bridge
    bridge_ports eth0
    bridge_stp no
    bridge_fd 2

restart your server or stop and start networking, if you’re on a local console

If you use bridging this way, nothing needs to be changed in xend config

non-local networking

I cannot help you out here - this usually goes too deep. However, I can give you some hints:

Networking can be really frustrating, especially if you have multiple bridges or public ip Addresses. Hosters often have special networking setups that require special actions, like setting up routes on the Dom0 or asking the hosters support to allow multiple mac addresses on a switchport.

The most important thing here is: Check if your hoster supports XEN before ordering!

Ideally there is a faq/bulletin that describes Virtual Machine Networking setup.

If you still have heavy problems with networking, I’am available for rent ;)

Paravirtualized DomUs

Usually I am not setting up just one DomU, but a lot. As every sysadmin is a lazy bastard (at least I am), I try to keep my systems as homogeneous as possible: same distribution, same standard packages, same configuration, etc. For example: You have a Mail gateway in your LAN. Why not pass this as relayserver to every DomU’s mailserver in the moment of creation? Or what about granting remote access by auto-providing your ssh public key?

Preparing for many DomUs

Note: this is a rather old fashioned way of auto-provisioning virtual servers and services. But it works pretty good. If you prefer the hot stuff, have a look at Chef!

I prefer installation with xen-tools, a toolset for semi-automatic DomU creation:

# sudo aptitude install xen-tools

In order to install an ubuntu release as domU, the corresponding folder must be existent in /usr/lib/xen-tools. Precise is not there, so we just copy the karmic folder:

# cp -a /usr/lib/xen-tools/karmic.d /usr/lib/xen-tools/precise.d

Why not symlink? Because you could add release-specific changes to the installation recipe like changing the default postfix configuration as described above, or installing toolsets and monitoring stuff like nagios-nrpe or munin-node.

You may also create a tar package and preinstall everything you need. Xen tools can handle tar-templates as well.

Now it’s time to create the DomU.

Create a DomU

# xen-create-image --bridge=xenbr0 --lvm=vg0 --dist=precise --fs=xfs --netmask=255.255.255.0 --gateway=10.0.0.254 --size=10Gb --swap=2Gb --memory=512Mb --ip=10.0.0.2 --hostname=myfirstdomU

If you always use the same parameters for your machines, I recommend to put them as default in /etc/xen-tools/xen-tools.conf

Now, rename the config file:

# mv /etc/xen/myfirstdomU.cfg /etc/xen/myfirstdomU

Reason: the config file name is now the same as the DomU name, so you can start/restart/stop with the same command.

Start it with

# xm create myfirstdomU

DomU control commands

start a domU:

# xm create <name> 

send a shutdown signal to the DomU:

# xm shut <name> 

Sudden death to the DomU. Same as taking away power - no proper shutdown. Use this only when DomU is not responding on the console:

# xm destroy <name> 

Fully virtualized DomUs (HVM) (tested with Windows7, and Windows Server 2008 R2)

If you need HVM DomUs (For Linux Systems, please use paravirtualized DomUs!), You can do the setup manually:

Prerequisites

Create the volumes you want to use, i.E. a 60Gb Disk:

# lvcreate -L 60G -n hvmdomu-disk /dev/vg0

and provision the installation iso image on the DomU, in my case this is /tmp/InstallImage.iso

Now, use the Ubuntu provided hvm configuration:

# zcat /usr/share/doc/xen-utils-common/examples/xmexample.hvm.gz > /etc/xen/hvmdomu

Edit your new DomU config file and enter what you just created (only changes listed, leave the rest as provided):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

[...]
name = 'hvmdomu'

vif = [ 'type=ioemu, bridge=xenbr0' ]

disk =  [
                'phy:/dev/vg0/hvmdomu-disk,hda,w',
                'file:/tmp/InstallImage.iso,hdc:cdrom,r'
        ]


boot="dc"
#change this to "cd" after installation!

vnc=1
vnclisten=0.0.0.0
vncunused=1
vncpasswd='supersecret'
# you may use a different one ;)


[...]

start the domu,

# xm create hvmdomu

connect with a VNC-viewer to your Dom0, Port 5900 (the next HVM DomU will bind to Port 5901, 5902, and so on) and perform the installation.

After the HVM System has installed its own bootloader (usually when it requests the first reboot), change the boot sequence in your config file as commented above. Yoy may also comment out the iso image once everything is set up.

That’s it.

Best practice hints

• instead of xm create/shutdown/console, use abbreviations: xm crea/shut/con
• When using Ubuntu 12.04 both as Dom0 and DomU, hot adding and removing memory works out of the box. Nice!
• When using xfs as filesystem, growing Disk size without downtime is possible, too!

If memory increasing does not have any effect, check, if the menory is present, but not registered:

# grep offline /sys/devices/system/memory/*/state

note the numbers and activate them one by one:

# echo online > /sys/devices/system/memory/memory[number]/state

starting DomUs on Dom0 startup

all DomUs that are present in /etc/xen/auto will be started directly after system startup:

# mkdir /etc/xen/auto
# cd /etc/xen/auto
# ln -s ../<name> .

Note that we have set

XENDOMAINS_RESTORE=false

in /etc/default/xendomains!

Troubeshooting DomUs

Perform these tasks on the Dom0:

This will start the DomU with attached console and lets you view the booting process. If this hangs, check the kernel messages. Exit the console with “ctrl + ]”

# xm create -c myfirstdomU

This will attach to the console of an already started DomU. When networking is not working, you still can act on the local console to perform some commands. Exit the console with “ctrl + ]”

# xm console myfirstdomU

There is no console on HVM DomUs, use VNC for diagnostics.

I don’t need to explain ping, do I? If this is not working, check the networking setup:

# ping <DomU IP> 

coming up next:

DomU instant cloning and backup with lvm snapshots (THE perfect solution for test/staging systems)

recipe based on:

• https://help.ubuntu.com/community/XenProposed
• http://wiki.debian.org/Xen
• several years playing around with XEN :)