Introduction
This page describes several procedures which Alexis Huxley uses to configure and test cluster services on his network.
Procedures
Installing cluster software and configuring keys
This procedure is based on configure_cluster_server() in pdi. It may be converted properly later if it is required.
- Run:
loading configure_cluster_server configure_cluster_server() { local NWADDR IPADDR NETMASK MCADDR ROOT_PW # Get settings we'll need for this task. get_setting CLUSTERNODE_FLAG || return $? ! $CLUSTERNODE_FLAG || { get_setting CLUSTERNODE_NIC || return $? get_setting UNAME || return $? get_setting RELEASE || return $? get_setting NIC_CONFIG_IPADDR_$(nic2hashindex $CLUSTERNODE_NIC) || return $? get_setting NIC_CONFIG_NETMASK_$(nic2hashindex $CLUSTERNODE_NIC) || return $? } [ $TASK_MODE != get_settings ] || return 0 # Task applicability $CLUSTERNODE_FLAG || return 0 # Sanity checks and derivations if [ $RELEASE = lenny ]; then error "$RELEASE: does not have pacemaker or pacemaker-compatible hb_gui" elif [ $RELEASE = squeeze ]; then : else internal "configure_cluster_server: $RELEASE: unhandled" fi IPADDR=$(ifconfig $CLUSTERNODE_NIC | sed -n 's/.*inet addr:\([^ ]*\).*/\1/p') NETMASK=$(ifconfig $CLUSTERNODE_NIC | sed -n 's/.*Mask:\([^ ]*\).*/\1/p') NWADDR=$(ipaddrander $IPADDR $NETMASK) MCADDR=224.1.$((RANDOM % 256)).$((RANDOM % 256)) debug 10 "configure_cluster_server: IPADDR=$IPADDR, NETMASK=$NETMASK, NWADDR=$NWADDR, MCADDR=$MCADDR" debug 10 "configure_cluster_server: installing packages ..." pkgadmin install pacemaker pacemaker-mgmt-client pacemaker-mgmt || return $? # Set hacluster password to same as root. ROOT_PW=$(sed -n "s/^root:\([^:]*\):.*/\1/p" /etc/shadow) ROOT_PW=${ROOT_PW//\//\\\/} ROOT_PW=${ROOT_PW//$/\\$} perl -pi -e "s/^(hacluster:)([^:]*)(:.*)/\$1$ROOT_PW\$3/" /etc/shadow # All cluster nodes need to have some files the same; the # first-installed node sets up the files and the later-installed # nodes get copies of them. The simplest way to do this is to # generate the files on *all* nodes but then overwrite then on # the later-installed nodes. # Remove old key to ensure ssh-keygen doesn't ask about overwriting debug 10 "configure_cluster_server: generating key pair ..." rm -f ~/.ssh/id_dsa ~/.ssh/id_dsa.pub ssh-keygen -q -N "" -t dsa -f ~/.ssh/id_dsa # Make the machine trust itself (and if ~/.ssh is copied to the other # cluster nodes then it will make them trust each other). cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys # Generate the key for intra-cluster communications # corosync-keygen reads from /dev/random and that is ridculously slow # to provide sufficient random data. Unfortunately we can't tell it # to use /dev/urandom instead. # to use rm -f /dev/random mknod -m 666 /dev/random c 1 9 corosync-keygen # Slurp any chars we typed for corosync-keygen which it left in the input # buffer for whoever reads next. while read -n 1 -t 0.001 X; do :; done # Put things back the way they were. rm -f /dev/random mknod -m 666 /dev/random c 1 8 # Configure corosync perl -0777 -pi -e "s/^([ \\t]*)(interface {\\n).*?\\n([ \\t]*)(}\\n)/\$1\$2\$1 ringnumber: 0\\n\$1 bindnetaddr: $NWADDR\\n\$1 mcastaddr: $MCADDR\\n\$1 mcastport: 5405\\n\$3\$4/ms" /etc/corosync/corosync.conf perl -0777 -pi -e "s/^([ \\t]*)(service {\\n).*?\\n([ \\t]*)(}\\n)/\$1\$2\$1 ver: 0\\n\$1 name: pacemaker\\n\$1 use_mgmtd: 1\\n\$3\$4/ms" /etc/corosync/corosync.conf echo "START=yes" > /etc/default/corosync # Copy stuff to second node (which, if this is not the first node, will # mean overwriting stuff just created) send_email <<EOF Subject: $UNAME: last steps to configure this machine as an install server To: $MAILTO If this is not the first-installed node in this cluster then you should log in to a trust server and run: FIRST_NODE=<first-installed-node> LAST_NODE=$UNAME ssh \$FIRST_NODE "cd / && find /root/.ssh /etc/corosync/authkey /etc/corosync/corosync.conf /etc/default/corosync | cpio -o" | \ ssh \$LAST_NODE "cd / && cpio -iudm" EOF # Don't bother starting corosync; it can be left for the reboot. return 0 }
Configuring a VM to access multiple bridges
In VM servers, pdi (see MDI) can configure 3 bridges, each connected to a different VLAN and make them available to VMs. But the VM configuration still needs to be updated to make use of them.
- Run:
virsh shutdown <this-vm> virsh dumpxml <this-vm> > <this-vm>.xml
- Edit the XML file, clone the NIC stanza twice, incrementing the MAC address, bridge name and PCI slot in the clones, making sure that the original NIC stanza is not changed and that the PCI slot does not clash with any already present! E.g. If the original stanza was this:
<interface type='bridge'> <mac address='00:16:3e:dd:54:cf'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface>
and PCI slot numbers 0x03 and 0x04 were used by other stanzas then you would add this:
<interface type='bridge'> <mac address='00:16:3e:dd:54:d0'/> <source bridge='br1'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> <interface type='bridge'> <mac address='00:16:3e:dd:54:d1'/> <source bridge='br2'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </interface>
- Run:
virsh undefine <this-vm> virsh define <this-vm>.xml
- Run:
virsh start <this-vm>
- libvirt or libvirt-tools has a bug whereby XML configuration data for multiple NICs overwrites the XML configuration data for the first NIC, leading on the first edit to the impression that there is only one NIC and then on the second edit to there really being only one NIC. For this reason it is a good idea to preserve the XML files used above.
Tweaking basic cluster settings
This section lists various steps which may be needed; review them carefully to decide whether they are appropriate.
- Set the Unix password for the ‘hacluster’ account (this will be needed when using hb_gui).
- Disable STONITH (taken from https://wiki.clusterlabs.org/wiki/Debian_Lenny_HowTo), fix two-node quorum issues (taken from here) and make sure that resources do not migrate back by running:
noodle# crm crm(live)# cib new configtmp INFO: building help index INFO: configtmp shadow CIB created crm(configtmp)# configure crm(configtmp)configure# property stonith-enabled=false crm(configtmp)configure# property no-quorum-policy=ignore crm(configtmp)configure# rsc_defaults resource-stickiness=100 crm(configtmp)configure# verify crm(configtmp)configure# end There are changes pending. Do you want to commit them? y crm(configtmp)# cib use live crm(live)# cib commit configtmp INFO: commited 'configtmp' shadow CIB to the cluster crm(live)# cib delete configtmp crm(live)# quit
Testing using a dummy resource
- Set up a dummy resource by running:
noodle# crm crm(live)# cib new configtmp INFO: building help index INFO: configtmp shadow CIB created crm(configtmp)# configure crm(configtmp)configure# primitive dummy ocf:pacemaker:Dummy op monitor interval=10s WARNING: dummy: default timeout 20s for start is smaller than the advised 90 WARNING: dummy: default timeout 20s for stop is smaller than the advised 100 crm(configtmp)configure# verify WARNING: dummy: default timeout 20s for start is smaller than the advised 90 WARNING: dummy: default timeout 20s for stop is smaller than the advised 100 crm(configtmp)configure# end There are changes pending. Do you want to commit them? y crm(configtmp)# cib use live crm(live)# cib commit configtmp INFO: commited 'configtmp' shadow CIB to the cluster crm(live)# cib delete configtmp INFO: configtmp shadow CIB deleted crm(live)# quit bye noodle#
- Test by running the following commands (based on https://wiki.clusterlabs.org/wiki/Debian_Lenny_HowTo):
root# crm crm(live)# configure show node doodle \ attributes standby="off" node noodle \ attributes standby="off" primitive dummy ocf:pacemaker:Dummy \ op monitor interval="10s" property $id="cib-bootstrap-options" \ dc-version="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b" \ cluster-infrastructure="openais" \ expected-quorum-votes="2" \ stonith-enabled="false" \ no-quorum-policy="ignore" \ maintenance-mode="false" \ last-lrm-refresh="1291797358" rsc_defaults $id="rsc-options" \ resource-stickiness="100" op_defaults $id="op_defaults-options" \ record-pending="false" crm(live)# node show doodle: normal standby: off noodle: normal standby: off crm(live)# resource show dummy (ocf::pacemaker:Dummy) Started crm(live)# node standby <node-name> # verify resource is migrated to other node with "crm_mon -1" crm(live)# node online <node-name> # verify resource is note migrated back to other node with "crm_mon -1" crm(live)# resource migrate dummy <node-name> # verify resource is migrated with "crm_mon -1" crm(live)# resource stop dummy # verify resource is stopped with "crm_mon -1" crm(live)# resource start dummy # verify resource is started with "crm_mon -1" crm(live)# quit bye noodle# - Remove the dummy resource by running:
noodle# crm crm(live)# cib new configtmp INFO: building help index INFO: configtmp shadow CIB created crm(configtmp)# configure crm(configtmp)configure# delete dummy INFO: hanging location:cli-prefer-dummy deleted crm(configtmp)configure# verify crm(configtmp)configure# end There are changes pending. Do you want to commit them? y crm(configtmp)# cib use live crm(live)# cib commit configtmp INFO: commited 'configtmp' shadow CIB to the cluster crm(live)# cib delete configtmp INFO: configtmp shadow CIB deleted crm(live)# quit bye noodle#
Clustering Apache
- On all nodes install apache2
- On all nodes prevent automatic startup:
service apache2 stop update-rc.d apache2 remove
- On all nodes configure apache to listen on an as-yet-unconfigured virtual interface:
perl -pi -e 's/^Listen.*/Listen 192.168.1.13:80/' /etc/apache2/ports.conf
- On NFS shared storage (e.g. NAS) allocate storage to be accessible to both nodes
- On one node manually start resources to test understanding of what is required and in what order. E.g.:
mount storage.pasta.net:/vol/webpages /var/www ifconfig eth0:1 192.168.1.13 up service apache2 start
and check web access on the virtual interface.
- Manually stop resources.
- Add a resource group containing 3 resources for this service (vNIC, mount, apache). The resulting resources looked like this:
noodle# cibadmin -Q -o resources > resources.xml noodle# cat resources.xml <resources> <group id="webservices"> <meta_attributes id="webservices-meta_attributes"> <nvpair id="webservices-meta_attributes-target-role" name="target-role" value="started"/> </meta_attributes> <primitive id="vnic" provider="heartbeat" type="IPaddr2"> <operations id="vnic-operations"> <op id="vnic-op-monitor-10s" interval="10s" name="monitor" timeout="20s"/> </operations> <instance_attributes id="vnic-instance_attributes"> <nvpair id="vnic-instance_attributes-ip" name="ip" value="192.168.1.13"/> <nvpair id="vnic-instance_attributes-nic" name="nic" value="eth0:1"/> </instance_attributes> <meta_attributes id="vnic-meta_attributes"> <nvpair id="vnic-meta_attributes-target-role" name="target-role" value="started"/> </meta_attributes> </primitive> <primitive id="mount" provider="heartbeat" type="Filesystem"> <operations id="mount-operations"> <op id="mount-op-monitor-20" interval="20" name="monitor" timeout="40"/> </operations> <instance_attributes id="mount-instance_attributes"> <nvpair id="mount-instance_attributes-device" name="device" value="storage.pasta.net:/vol/www"/> <nvpair id="mount-instance_attributes-directory" name="directory" value="/var/www"/> </instance_attributes> <meta_attributes id="mount-meta_attributes"> <nvpair id="mount-meta_attributes-target-role" name="target-role" value="started"/> </meta_attributes> </primitive> <primitive id="apache2" type="apache2"> <operations id="apache2-operations"> <op id="apache2-op-monitor-15" interval="15" name="monitor" start-delay="15" timeout="15"/> </operations> </primitive> </group> </resources> noodle#This could be reloaded with:
cibadmin --replace --scope resources --xml-file resources.xml
Clustering Icinga
- On all nodes install icinga
- Work around BTS#599555 by creating XXXX containing the following (with hostname adjusted):
<VirtualHost *:80> ServerName icinga.pasta.net ServerAlias www.icinga.pasta.net DocumentRoot /usr/share/icinga/htdocs ScriptAlias /cgi-bin/icinga /usr/lib/cgi-bin/icinga # Where the stylesheets (config files) reside Alias /stylesheets /etc/icinga/stylesheets <Directory /usr/share/icinga/htdocs> Options FollowSymLinks Order allow,deny Allow from all </Directory> ErrorLog ${APACHE_LOG_DIR}/icinga.error.log CustomLog ${APACHE_LOG_DIR}/icinga.access.log combined </VirtualHost> - Run:
/etc/init.d/apache2 reload
- In /etc/apache2/conf.d/icinga, locate the specification of the htpasswd.users file.
- Use htpasswd to add an entry to that file.
At this point, I could access the Icinga tactical interface, but found it similar enough to Nagios that I did not want to continue.