In this post I will demonstrate how I am setting up rather big infrastructures (> 10 DomUs, >2 Dom0s)
First, I create a bridged network interface for my DomUs. In this case, it is a Dom0 in a private LAN.
If this is not yet installed, install the bridging utilities:
# sudo aptitude install bridge-utils
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
restart your server or stop and start networking, if you’re on a local console
If you use bridging this way, nothing needs to be changed in xend config
I cannot help you out here - this usually goes too deep. However, I can give you some hints:
Networking can be really frustrating, especially if you have multiple bridges or public ip Addresses. Hosters often have special networking setups that require special actions, like setting up routes on the Dom0 or asking the hosters support to allow multiple mac addresses on a switchport.
The most important thing here is: Check if your hoster supports XEN before ordering!
Ideally there is a faq/bulletin that describes Virtual Machine Networking setup.
If you still have heavy problems with networking, I’am available for rent ;)
Usually I am not setting up just one DomU, but a lot. As every sysadmin is a lazy bastard (at least I am), I try to keep my systems as homogeneous as possible: same distribution, same standard packages, same configuration, etc. For example: You have a Mail gateway in your LAN. Why not pass this as relayserver to every DomU’s mailserver in the moment of creation? Or what about granting remote access by auto-providing your ssh public key?
Preparing for many DomUs
Note: this is a rather old fashioned way of auto-provisioning virtual servers and services. But it works pretty good. If you prefer the hot stuff, have a look at Chef!
I prefer installation with xen-tools, a toolset for semi-automatic DomU creation:
# sudo aptitude install xen-tools
In order to install an ubuntu release as domU, the corresponding folder must be existent in /usr/lib/xen-tools. Precise is not there, so we just copy the karmic folder:
# cp -a /usr/lib/xen-tools/karmic.d /usr/lib/xen-tools/precise.d
Why not symlink? Because you could add release-specific changes to the installation recipe like changing the default postfix configuration as described above, or installing toolsets and monitoring stuff like nagios-nrpe or munin-node.
You may also create a tar package and preinstall everything you need. Xen tools can handle tar-templates as well.
Now it’s time to create the DomU.
Create a DomU
# xen-create-image --bridge=xenbr0 --lvm=vg0 --dist=precise --fs=xfs --netmask=255.255.255.0 --gateway=10.0.0.254 --size=10Gb --swap=2Gb --memory=512Mb --ip=10.0.0.2 --hostname=myfirstdomU
If you always use the same parameters for your machines, I recommend to put them as default in /etc/xen-tools/xen-tools.conf
Now, rename the config file:
# mv /etc/xen/myfirstdomU.cfg /etc/xen/myfirstdomU
Reason: the config file name is now the same as the DomU name, so you can start/restart/stop with the same command.
Start it with
# xm create myfirstdomU
DomU control commands
start a domU:
# xm create <name>
send a shutdown signal to the DomU:
# xm shut <name>
Sudden death to the DomU. Same as taking away power - no proper shutdown. Use this only when DomU is not responding on the console:
# xm destroy <name>
Fully virtualized DomUs (HVM) (tested with Windows7, and Windows Server 2008 R2)
If you need HVM DomUs (For Linux Systems, please use paravirtualized DomUs!), You can do the setup manually:
Create the volumes you want to use, i.E. a 60Gb Disk:
# lvcreate -L 60G -n hvmdomu-disk /dev/vg0
and provision the installation iso image on the DomU, in my case this is /tmp/InstallImage.iso
Now, use the Ubuntu provided hvm configuration:
# zcat /usr/share/doc/xen-utils-common/examples/xmexample.hvm.gz > /etc/xen/hvmdomu
Edit your new DomU config file and enter what you just created (only changes listed, leave the rest as provided):
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
start the domu,
# xm create hvmdomu
connect with a VNC-viewer to your Dom0, Port 5900 (the next HVM DomU will bind to Port 5901, 5902, and so on) and perform the installation.
After the HVM System has installed its own bootloader (usually when it requests the first reboot), change the boot sequence in your config file as commented above. Yoy may also comment out the iso image once everything is set up.
Best practice hints
- instead of xm create/shutdown/console, use abbreviations: xm crea/shut/con
- When using Ubuntu 12.04 both as Dom0 and DomU, hot adding and removing memory works out of the box. Nice!
- When using xfs as filesystem, growing Disk size without downtime is possible, too!
If memory increasing does not have any effect, check, if the menory is present, but not registered:
# grep offline /sys/devices/system/memory/*/state
note the numbers and activate them one by one:
# echo online > /sys/devices/system/memory/memory[number]/state
starting DomUs on Dom0 startup
all DomUs that are present in /etc/xen/auto will be started directly after system startup:
# mkdir /etc/xen/auto # cd /etc/xen/auto # ln -s ../<name> .
Note that we have set
Perform these tasks on the Dom0:
This will start the DomU with attached console and lets you view the booting process. If this hangs, check the kernel messages. Exit the console with “ctrl + ]”
# xm create -c myfirstdomU
This will attach to the console of an already started DomU. When networking is not working, you still can act on the local console to perform some commands. Exit the console with “ctrl + ]”
# xm console myfirstdomU
There is no console on HVM DomUs, use VNC for diagnostics.
I don’t need to explain ping, do I? If this is not working, check the networking setup:
# ping <DomU IP>
coming up next:
DomU instant cloning and backup with lvm snapshots (THE perfect solution for test/staging systems)
recipe based on:
- several years playing around with XEN :)