Configuring virtualisation services (revision 4)

Warning: development of this configuration was abandonned due to:

  1. lack of good synchronisation tools
  2. some ZFS issues

Introduction

This page describes how Alexis Huxley installed and configures a virtualisation environment providing KVM-based VMs and LXC-based containers and replicated VM/container images. This procedure is intended for Debian 12.

Note that NAS services are provided by a KVM-based VM and are documented elsewhere.

Hardware

  • two systems are required
  • RAID1 is not needed (redundancy will be provided by ZFS replication)
  • optionally RAID0 over multiple disks in each server (RAID0 will help improve IO speeds; it will be provided by ZFS)
  • two NICs are required in each host

Prerequisite state

This section is here to make explicit that the OSs shall be configured to use ZFS.

Local storage

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. Create volumes: 
    zfs create -o mountpoint=legacy -o quota=200g zpool0/local
  3. Add fstab entries for them all as below, create mountpoints and mount them: 
    zpool0/local /vol/local zfs auto,noatime,nodiratime 0 2

Replicated storage

  1. Even when using ZFS, it’s likely that LVM is still installed. By default, LVM finds PVs inside VM images. To prevent this edit /etc/lvm/lvm.conf and set: 
    devices {
    ...
    filter = [ "r|/dev/drbd.*|", "r|/dev/vg.*|", "r|/dev/zd.*|" ]
    ...
}

    (It is not necessary to reboot after that edit.)

  2. Something needs to go here once I have written zfsadm!

Shared public network interface

The VMs that run on each node will need access to the public network interface.

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. Reconfigure the stanza for the first NIC in /etc/network/interfaces accordingly. E.g.: 
    iface eth0 inet manual

auto br0
iface br0 inet static
    address 192.168.1.6
    netmask 255.255.255.0
    gateway 192.168.1.1
    bridge_ports <public-interface>

Dedicated network interface for cluster communications

It is essential to use a dedicated network card for cluster communications in order to ensure that public traffic does not impact replication.

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. Add a suitable entry to /etc/network/interfaces for the NIC you will use for the cluster communications and add an entry for it to /etc/hosts. E.g.: 
    auto <replication-interface>
iface eth1 inet static
    address 192.168.3.6
    netmask 255.255.255.0
  3. Reboot.

Hypervisors

This procedure is to be run on both nodes, regardless of whether they are both being configured at the same time or not, unless explicitly stated otherwise.

  1. If the system was installed with PCMS then skip this section as it will already have been done by PCMS.
  2. Run: 
    #  basic libvirt infrastructure
apt -y install uuid-runtime netcat-openbsd dnsmasq libvirt-daemon \
               libvirt-daemon-system libvirt-clients
#  KVM/QEMU support
apt -y install qemu-system-x86 qemu-utils ovmf
#  LXC support
apt -y install lxc lxcfs libpam-cgfs lxc-templates libfile-lchown-perl \
               libvirt-daemon-driver-lxc xmlstarlet debootstrap \
               distro-info

    and reboot.

Workaround for load average leaking from host to container

I’m not yet certain this works, but it’s what I read here.

  1. Run: 
    cp /lib/systemd/system/lxcfs.service /etc/systemd/system/

    (libpam-cgfs is mentioned in /usr/share/doc/lxcfs/README.Debian.)

  2. Edit /etc/systemd/system/lxcfs.service and change: 
    ExecStart=/usr/bin/lxcfs --enable-loadavg /var/lib/lxcfs
  3. Run: 
    systemctl daemon-reload
systemctl restart lxcfs
  4. On Debian 12 this did not work for me, even after a reboot.

Hypervisor plugins

In order to support mounting disk images on unprivileged containers, a plugin is needed.

  1. Run: 
    mkdir -p ~/opt/
svn co -q https://svn.pasta.freemyip.com/main/virttools/trunk ~/opt/virttools
mkdir -p /etc/libvirt/hooks/lxc.d
ln ~/opt/virttools/bin/etc-libvirt-hooks-lxc.d-mounter /etc/libvirt/hooks/lxc.d/mounter

    (Notice that ln command does not use the -s option.)

Virtual resources

  1. Create and define the local storage pool: 
    mkdir /vol/local/images
virsh pool-define-as --name=local   --type=dir \
    --target=/vol/local/images
virsh pool-start local
virsh pool-autostart local
  2. Copy any ISO images you might need into that pool and then run: 
    virsh pool-refresh local
  3. Remove pre-defined but unwanted networks: 
    virsh net-destroy default
virsh net-undefine default
  4. Since we plumb VMs’ NICs into the sharable br0 bridge and br0 is not managed by libvirt there is nothing to do at this time to configure access to the public network.
  5. Define a network to allow co-hosted VMs to communicate with each other directly, e.g.: 
    virsh net-define <(cat <<EOF
<network>
  <name>192.168.10.0</name>
  <uuid>$(uuidgen)</uuid>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:81:cd:08'/>
  <ip address='192.168.10.1' netmask='255.255.255.0'>
  </ip>
</network>
EOF
)
virsh net-autostart 192.168.10.0
virsh net-start 192.168.10.0
  6. Set up SSH keys to allow the running of virt-manager from a remote system.

Tips & tricks

  1. To connect to VM’s graphical console it is possible to use something like this: 
    virt-viewer --connect=qemu+ssh://root@torchio/system lagane
 # or set LIBVIRT_DEFAULT_URI and drop the '--connect=...'

See also