Configuring virtualisation services (revision 5)

Introduction

This page describes how Alexis Huxley installed and configures a virtualisation environment providing KVM-based VMs and LXC-based containers and replicated VM/container images. This procedure is intended for Debian 12.

Note that NAS services are provided by a KVM-based VM and are documented elsewhere.

Hardware

  • two systems are required
  • optionally RAID0 over multiple disks in each server to improve disk access speeds
  • two NICs are required in each host (one for host access and one for replication)

Local storage

Virtualisation servers will use DRBD-replicated storage for most VMs. However, occasionally, local space is useful (e.g. for a test VM, for snapshotting DRBD devices).

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. If the systems are pestaroli and testaroli then they may not local storage and then skip this section.
  3. Create LVs: 
    lvcreate --name=local --size=200g vg0
  4. Format for XFS, which offers online size changing: 
    mkfs -t ext4 -f /dev/vg0/local
  5. Add fstab entries for them all as below, create mountpoints and mount them: 
    /dev/mapper/vg0-local /vol/local ext4 auto,noatime,nodiratime 0 2

Replicated storage

  1. Run: 
    apt-get -y install drbd-utils
  2. Even when using ZFS, it’s likely that LVM is still installed. By default, LVM finds PVs inside VM images. To prevent this edit /etc/lvm/lvm.conf and set: 
    devices {
    ...
    filter = [ "r|/dev/drbd.*|", "r|/dev/vg.*|", "r|/dev/zd.*|" ]
    ...
}

    (It is not necessary to reboot after that edit.)

Shared public network interface

The VMs that run on each node will need access to the public network interface.

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. Reconfigure the stanza for the first NIC in /etc/network/interfaces accordingly. E.g.: 
    iface eth0 inet manual

auto br0
iface br0 inet static
    address 192.168.1.10
    netmask 255.255.255.0
    gateway 192.168.1.1
    bridge_ports <public-interface>

Dedicated network interface for cluster communications

It is essential to use a dedicated network card for cluster communications in order to ensure that public traffic does not impact replication.

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. Add a suitable entry to /etc/network/interfaces for the NIC you will use for the cluster communications and add an entry for it to /etc/hosts. E.g.: 
    auto <replication-interface>
iface eth1 inet static
    address 192.168.3.10
    netmask 255.255.255.0
  3. Reboot.

Hypervisors

This procedure is to be run on both nodes, regardless of whether they are both being configured at the same time or not, unless explicitly stated otherwise.

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. Run: 
    #  basic libvirt infrastructure
apt -y install uuid-runtime netcat-openbsd dnsmasq libvirt-daemon \
               libvirt-daemon-system libvirt-clients
#  KVM/QEMU support
apt -y install qemu-system-x86 qemu-utils ovmf
#  LXC support
apt -y install lxc lxcfs libpam-cgfs lxc-templates libfile-lchown-perl \
               libvirt-daemon-driver-lxc xmlstarlet debootstrap \
               distro-info

    and reboot.

Workaround for load average leaking from host to container

I’m not yet certain this works, but it’s what I read here.

  1. Run: 
    cp /lib/systemd/system/lxcfs.service /etc/systemd/system/

    (libpam-cgfs is mentioned in /usr/share/doc/lxcfs/README.Debian.)

  2. Edit /etc/systemd/system/lxcfs.service and change: 
    ExecStart=/usr/bin/lxcfs --enable-loadavg /var/lib/lxcfs
  3. Run: 
    systemctl daemon-reload
systemctl restart lxcfs
  4. On Debian 12 this did not work for me, even after a reboot.

Hypervisor plugins

In order to support mounting disk images on unprivileged containers, a plugin is needed.

  1. Run: 
    mkdir -p ~/opt/
svn co -q https://svn.pasta.freemyip.com/main/virttools/trunk ~/opt/virttools
mkdir -p /etc/libvirt/hooks/lxc.d
ln ~/opt/virttools/bin/etc-libvirt-hooks-lxc.d-mounter /etc/libvirt/hooks/lxc.d/mounter

    (Notice that ln command does not use the -s option.)

Virtual resources

  1. Create and define the local storage pool: 
    mkdir /vol/local/images
virsh pool-define-as --name=local   --type=dir \
    --target=/vol/local/images
virsh pool-start local
virsh pool-autostart local
  2. Copy any ISO images you might need into that pool and then run: 
    virsh pool-refresh local
  3. Remove pre-defined but unwanted networks: 
    virsh net-destroy default
virsh net-undefine default
  4. Since we plumb VMs’ NICs into the sharable br0 bridge and br0 is not managed by libvirt there is nothing to do at this time to configure access to the public network.
  5. Define a network to allow co-hosted VMs to communicate with each other directly, e.g.: 
    virsh net-define <(cat <<EOF
<network>
  <name>192.168.10.0</name>
  <uuid>$(uuidgen)</uuid>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:81:cd:08'/>
  <ip address='192.168.10.1' netmask='255.255.255.0'>
  </ip>
</network>
EOF
)
virsh net-autostart 192.168.10.0
virsh net-start 192.168.10.0
  6. Set up SSH keys to allow the running of virt-manager from a remote system.

Tips & tricks

  1. To connect to VM’s graphical console it is possible to use something like this: 
    virt-viewer --connect=qemu+ssh://root@torchio/system lagane
 # or set LIBVIRT_DEFAULT_URI and drop the '--connect=...'

See also