Beowulf Installation Guide/VM Host Setup

This part of the Beowulf Installation Guide shows how to set up a virtual machine host that will house a cluster admin node as a guest. We will use KVM as the virtualization engine, with libvirt and related tools used to manage the virtual machines.

Background
Virtualization allows an entire virtual computer (called a virtual machine or VM) entirely inside software, itself inside a real machine. We house the admin nodes of our clusters inside virtual machines to make them more robust against failure. Among other things, we can:
 * Move the admin node's VM to another host machine. This can be useful if a hardware failure on the host machine is imminent, or if we want to perform hardware upgrades.
 * Clone the VM. This allows us to test software or OS upgrades, or even rebuild the system from scratch while another one is running. When the upgrade is finished, we can swap the new version in place of the old one with minimal downtime.
 * Run multiple VMs on the same machine. This way, we can run other servers, like the Subversion server, on the same hardware as an admin node without the machines having to be associated (by the network or account configuration, for example) in any way.
 * Efficiently back up entire hard drives. Although there are many ways to back up data (even complete hard disk images) without using virtualization, it does give us more options and flexibility.

KVM is a virtualization engine that is built-in to the Linux kernel. Many of its components are borrowed from QEMU, another virtualization suite. As a result, you will see references to QEMU in many of the commands we will use to manage VMs.

While the KVM and QEMU tools can be used to manage the virtual machines directly, we will make use of libvirt, a library that abstracts different virtualization technologies like KVM, Xen, and VMware into a unified API. There are many useful interfaces and tools that are built on top of this library, such as virsh (a part of the libvirt project itself) and Virtual Machine Manager, a graphical interface.

See the Wikipedia pages or the Resources section for a listing of useful sites relating to these virtualization tools.

CPU Virtualization Support
KVM uses hardware-assisted virtualization to achieve its high performance. Currently, this means that KVM requires a processor with either Intel's VT-x or AMD's AMD-V virtual machine extensions (VMXs). Practically all new processors come with this support, although many manufactured before the mid-2000s do not. On a working Linux system, you can find out if your processor has either of these features by running the command: If the command produces output, you have support.

Note that you may also have to enable VT-x or AMD-V in the computer's BIOS.

Device Passthrough
Another useful though not strictly required feature is device passthrough, which requires an IOMMU such as Intel's VT-d or AMD's AMD-Vi (not to be confused with VT-x or AMD-V, the CPU technologies). Both of these technologies are enabled in the chipset. Unfortunately, support for these technologies is still spotty as of mid-2010.

See the PCI Passthrough chapter in the Fedora Virtualization Guide for more information.

Install OS
We will use Ubuntu Server Edition (at version 10.04 LTS "Lucid Lynx" as of this writing) as the operating system for our virtual machine host. Download the CD image from the Ubuntu website and follow the instructions in the Ubuntu Server Guide on Installing from CD, choosing the following options along the way:
 * Use LVM for the partition layout, and only allocate as much as needed for the host system. In the Partition Disks step (see also LVM in the USG):
 * When it asks for a partitioning method, choose Guided - Use entire disk and set up LVM.
 * Choose the main disk to partition in the next step.
 * When it asks Write the changes to disks and configure LVM?, choose Yes.
 * When it asks for the Amount of volume group to use for guided partitioning, choose an amount of space that is sufficient for a basic OS installation plus swap space. A decent amount to start with is . You may need even less.
 * When it asks Write changes to disks?, choose Yes.
 * Since Ubuntu does not enable the root account by default, setup will create a user account with privileges to gain superuser access through sudo. You will need to share this login among all the administrators.
 * Name the account bw-admin.
 * Use a strong password agreed to by all administrators.


 * When asked How do you want to manage upgrades on this system?, choose Install security updates automatically.
 * In the Software Selection step, choose the following tasks:
 * OpenSSH Server
 * Virtual Machine host

After the installation is complete, remove the disk and boot into your new host operating system.

Setup Virtualization Software
If you did not already install KVM and libvirt by selecting the "Virtual machine host" task while installing the OS, install them now using the tasksel command:

Now install the virt-install package:

Finally, add the bw-admin user to the libvirtd group to allow that user to manage virtual machines on the system without using sudo:

Optional: Install virt-manager
The Virtual Machine Manager GUI application can make setting up and managing a VM much easier. If you want to install it, along with an X.org server and the Fluxbox window manager, install the following packages:

In this configuration, X11 will not be started automatically. You will have to start it after you login by issuing the command:

Once X is running, you can open a terminal (or press Alt + F2) and issue the virt-manager command to bring up the Virtual Machine Manager.

Optional: Setup remote management
libvirt has features that allow you to connect to a host machine remotely to manage the VMs running inside it. The easiest way is SSH tunneling. To make this work, you will need to set up an SSH server on the VM host. Install the OpenSSH Server task if you did not do so during setup:

If you want remote management to work with, you must also setup key-based SSH authentication from your account on the remote machine to bw-admin on the VM host. See Passwordless SSH setup for instructions.

Once you have remote management setup, and you have packages like libvirt and/or Virtual Machine Manager installed on the remote machine, you should be able to connect from the remote machine to your VM host with commands like:

You can now use these commands instead of connecting with  on the VM host itself.

Setup Network Bridging
For your virtual machine to function as a server on the network, other machines will have to see it. For this, we need to set up bridged networking, which will make it appear as though your virtual machine(s) are connected to the outside network through the same interface as your host machine.

Since cluster admin nodes need two interfaces—one facing the outside network, and another facing the cluster network—we will need to set up two bridges. If we want the host machine to appear on the network (which is desirable on the outside network), we assign the bridge device an IP address the same way we would normally assign one to the physical device (i.e., through DHCP or by configuring it statically). Since we do not want the host machine to be visible on the cluster-facing network, we will make that bridge "transparent" by not assigning it an IP address. The virtual machine receives its IP addresses in the normal way, as though it were on the networks themselves.

First, install the required packages:

Next, open the. We are going to remove the stanzas that deal with the two Ethernet interfaces,  and , and replace them with bridges named   and. You might see something like this: auto eth0 iface eth0 inet dhcp
 * 1) The primary (outside) interface

auto eth1 iface eth1 inet dhcp
 * 1) The cluster-facing interface


 * Note: If you do not see a second interface, don't worry. Setup may not have configured your cluster-facing interface, since it could not get an IP address from it. You can check that the kernel detected your interface by running:
 * If you see any output, your device should be detected.
 * If you see any output, your device should be detected.

Edit those two stanzas to look like this: auto br0 iface br0 inet dhcp bridge_ports eth0 bridge_fd 9 bridge_hello 2 bridge_maxage 12 bridge_stp off
 * 1) The outside-facing bridge:
 * 2) - Gets IP for host machine via DHCP
 * 3) - Bridges to eth0, the outside-facing interface

auto br1 iface br1 inet manual bridge_ports eth1 bridge_fd 9 bridge_hello 2 bridge_maxage 12 bridge_stp off
 * 1) The cluster-facing bridge:
 * 2) - No IP for host machine
 * 3) - Bridges to eth1, the cluster-facing interface

Note that we used  in the second interface. This tells the boot scripts that we will configure the interface (i.e., give it an IP address) later. Since we never do that, it effectively keeps the bridge from acquiring an address, making it transparent on the network. This is what we want.

Now, restart networking and check that the bridge was configured correctly:

For more information, see Networking on the libvirt wiki.

Setup a Virtual Machine
Now we will create the virtual machine for the admin node. Our goals for this are to create a system that has:
 * Two virtual hard drives. Physically, these will be LVM volumes on the host machine:
 * A small (10-15 GB) volume for the root and swap partitions. It should be large enough to comfortably contain the base system, but small enough that we can easily store backups of the entire disk image.
 * A larger volume to contain the /home partition. It should be as large as possible so that users can store plenty of data, but you should leave at least as much room as your smaller volume so you can make a "snapshot volume" for backup (described later). Additionally, you may want to leave enough space to duplicate the /home partition as well. On a 1 TB drive with 25 GB taken up by other volumes/partitions, we recommend making this volume about 450 GB.
 * Bridged networking as described in the previous section
 * A comfortable amount of RAM, but with room for the host system and possibly other VMs that run on this host. Use no more than half of the total RAM to start with. It is very easy to add more later.

These instructions will use shell commands like virsh. Using the Virtual Machine Manager GUI should be self-explanatory if you have set it up.

Add Volumes for Virtual Disks
First use vgscan to determine what your volume group is named: In this example, the volume group is named.

Next, we will create two logical volumes in that volume group: Replace VM_NAME with the name of your cluster and VOLUME_GROUP_NAME with the name determined in the previous step. Adjust the sizes as appropriate.

Create the VM
Use the virt-install command to create the VM: Replace these values:
 * VM_NAME: The name of your cluster
 * VOL_GROUP: The name of the volume group containing the logical volumes you created above
 * INSTALLER_CD: The path to the ISO of the Ubuntu Installer CD, or the path to your CD-ROM device if you have the physical disk inserted (e.g., )