Configuring storage services generation two

Introduction

This page describes several procedures which Alexis Huxley uses to set up storage services on his network.

The partitioning scheme detailed here involves:

root, swap on RAID1 metadevices
remaining storage on RAID1 or RAID5, depending on the number of disks available.
that remaining storage is allocated to one LVM2 volume group
that volume group is divvied up into LVM2 logical volumes
those logical volumes are mounted at /vol/<share-name> and exported via NFS or SMB, or are simply shared via iSCSI
NFS clients will mount NFS shares as /staging/<share-name>

The expected starting point for this procedure is to have a non-RAIDed system with partitions for root and swap and all remaining space being unallocated.

This procedure has been superceded by Configuring storage services generation three; it remains here for reference.

Procedures

Raiding root and swap (lenny)

Insert the second disk.

Use fdisk to partition the second disk as per the first disk. Set the bootable flag as per the first disk. Set the partition type to 0xfd (Linux raid) on all partitions. The fdisk session will run something like this:

torchio# fdisk -l /dev/sda
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1459    11719386   83  Linux
/dev/sda2            1460        1520      489982+  82  Linux swap / Solaris
torchio# fdisk /dev/sdb
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-38913, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-38913, default 38913): 1459

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (1460-38913, default 1460):
Using default value 1460
Last cylinder or +size or +sizeM or +sizeK (1460-38913, default 38913): 1520

Command (m for help): a
Partition number (1-4): 1

Command (m for help): t
Partition number (1-4): 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
torchio#

Create a RAID1 device for the OS containing only the appropriate partition from the second disk. E.g.:
```
mdadm --create --auto=yes /dev/md0 --level=1 --force --raid-devices=1 /dev/sdb1
```
Create a RAID1 device for swap containing only the appropriate partition from the second disk. E.g.:
```
mdadm --create --auto=yes /dev/md1 --level=1 --force --raid-devices=1 /dev/sdb2
```
Make a filesystem on the first RAID1 device:
```
mkfs -t ext3 /dev/md0
```
Make swap space on the second RAID1 device:
```
mkswap /dev/md1
```
The post-install script for the mdadm package has updated the initrd file to include drivers and tools for MD devices, but it will only do this for the initrd belonging to the kernel which was active when mdadm was installed. This means that, when trying to boot other kernels (e.g. the one for allowing use of VirtualBox) it will be impossible to use /dev/md0 as the root filesystem. Fix this by running:
```
{
    echo -e "DEVICE partitions\nMAILADDR root"
    mdadm --examine --scan | sed 's/$/ spares=0/'
} > /etc/mdadm/mdadm.conf
ls /boot/vmlinuz* | sed 's/.*\/vmlinuz-//' | while read KV; do
    update-initramfs -u -k $KV
done
```
(The creation of /etc/mdadm/mdadm.conf will prevent a warning from update-initramfs.) Then verify that /etc/mdadm/mdadm.conf contains no spurious lines (I saw it containing two entries for /dev/md1 following imperfect attempt to back out creation of a device.)
Copy the current root filesystem on to the RAID1 device. E.g.:
```
mount /dev/md0 /mnt
rsync -ax / /mnt/
```
(Don’t unmount /mnt just yet.)

Put grub on the first disk:

grub --batch
root (hd0,0)
setup (hd0)
quit

Modify /boot/grub/menu.lst on the second disk (i.e. edit /mnt/boot/grub/menu.lst) to reference the root MD device in the kernel command line. E.g.:
```
# kopt=root/dev/md0 ro
...
kernel .... root=/dev/md0
```
Modify /etc/fstab on the second disk (i.e. edit /mnt/etc/fstab) to reference the root and swap MD devices.
Unmount the RAID device:
```
umount /mnt
```
Reboot the system booting off the second hard disk; this probably requires typing something like the following to the grub prompt:
```
root (hd1,0)
kernel /boot/vmlinuz-XXXXX root=/dev/md0 ro
initrd /boot/initrd-XXXXX
```
Use fdisk to partition the first disk, leaving the partition sizes alone because they should already be correct (after all, the sizes were copied from this disk to the second disk), but set the type of each partition to 0xfd regardless of its type.
Modify the RAID device for the OS to add the appropriate partition from the first disk. E.g.:
```
mdadm /dev/md0 --add /dev/sda1
mdadm --grow /dev/md0 -n 2
```
Modify the RAID device for swap to add the appropriate partition from the first disk. E.g.:
```
mdadm /dev/md1 --add /dev/sda2
mdadm --grow /dev/md1 -n 2
```
Before doing the next step you must wait for the RAID syncing to finish (otherwise grub will not find what it is looking for). Wait for it to finish by running:
```
watch cat /proc/mdstat
```
Record information about the new meta-device so that mdadm can do certain operations without scanning the disks with the command:
```
{
    echo -e "DEVICE partitions\nMAILADDR root"
    mdadm --examine --scan | sed 's/$/ spares=0/'
} > /etc/mdadm/mdadm.conf
```
(If this step is done before the sync completes, then spurious ‘spares’ lines will appear in the config file.)
Use grub in batch mode to set up the second disk. E.g.:
```
grub --batch
root (hd1,0)
setup (hd1)
quit
```

Raiding root and swap (squeeze)

Insert the second disk.

Use parted to partition the second disk as per the first disk. Set the boot flag as per the first disk. Set the raid flag on all partitions. The parted session will run something like this:

#  Do everything in bytes
(parted) units byte      
#  Select first disk to read sizes
(parted) select /dev/sda 
#  Read sizes
(parted) print

Number  Start         End           Size          Type     File system     Flags
 1      1048576B      11999903743B  11998855168B  primary  ext3            boot
 2      11999903744B  12500074495B  500170752B    primary  linux-swap(v1)

#  Switch to second disk to write sizes
(parted) select /dev/sdb
#  Create partitions on second disk matching partitions on first dis
(parted) mkpart primary ext3 1048576B 11999903743B
(parted) mkpart primary linux-swap 11999903744B  12500074495B
#  Set boot flag to match first disk, set raid mode
(parted) set 1 boot on
(parted) set 1 raid on
(parted) set 2 raid on
#  Display table to verify
(parted) print

Number  Start         End           Size          Type     File system  Flags
 1      1048576B      11999903743B  11998855168B  primary               boot, raid
 2      11999903744B  12500074495B  500170752B    primary               raid

(parted) quit

Create a RAID1 device for the OS containing only the appropriate partition from the second disk. E.g.:
```
mdadm --create --auto=yes /dev/md0 --level=1 --force --raid-devices=1 /dev/sdb1
```
Create a RAID1 device for swap containing only the appropriate partition from the second disk. E.g.:
```
mdadm --create --auto=yes /dev/md1 --level=1 --force --raid-devices=1 /dev/sdb2
```
Make a filesystem on the first RAID1 device:
```
mkfs -t ext3 /dev/md0
```
Make swap space on the second RAID1 device:
```
mkswap /dev/md1
```
The post-install script for the mdadm package has updated the initrd file to include drivers and tools for MD devices, but it will only do this for the initrd belonging to the kernel which was active when mdadm was installed. This means that, when trying to boot other kernels (e.g. the one for allowing use of VirtualBox) it will be impossible to use /dev/md0 as the root filesystem. Fix this by running:
```
{
    echo -e "DEVICE partitions\nMAILADDR root"
    mdadm --examine --scan | sed 's/$/ spares=0/'
} > /etc/mdadm/mdadm.conf
ls /boot/vmlinuz* | sed 's/.*\/vmlinuz-//' | while read KV; do
    update-initramfs -u -k $KV
done
```
(The creation of /etc/mdadm/mdadm.conf will prevent a warning from update-initramfs.) Then verify that /etc/mdadm/mdadm.conf contains no spurious lines (I saw it containing two entries for /dev/md1 following imperfect attempt to back out creation of a device.)
Copy the current root filesystem on to the RAID1 device. E.g.:
```
mount /dev/md0 /mnt
rsync -ax / /mnt/
```
(Don’t unmount /mnt just yet and do wait for this command to finish before proceeding.)
Put the grub loader on the first disk’s MBR, supplementing it with the necessary grub modules to make it intelligent enough to boot with /dev/md0 as root:
```
grub-install --modules="raid mdraid part_msdos ext2 linux normal help" \
    --root-directory=/mnt /dev/sda
```
Modify /etc/fstab on the second disk (i.e. edit /mnt/etc/fstab), changing the references to /dev/sda1 and /dev/sda2 to /dev/md0 and /dev/md1 respectively.
Unmount the RAID device:
```
umount /mnt
```

Reboot the system booting off the second hard disk by typing the following to the grub prompt:

set root='(hd0,msdos1)'
linux /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md0 ro nolapic
initrd /boot/initrd-2.6.32-5-amd64
boot

Use parted to partition the first disk, leaving the partition sizes alone because they should already be correct (after all, the sizes were copied from this disk to the second disk), but setting flags as below:
```
parted /dev/sda
set 1 raid on
set 2 raid on
quit
```
Modify the RAID device for the OS to add the appropriate partition from the first disk. E.g.:
```
mdadm /dev/md0 --add /dev/sda1
mdadm --grow /dev/md0 -n 2
```
Modify the RAID device for swap to add the appropriate partition from the first disk. E.g.:
```
mdadm /dev/md1 --add /dev/sda2
mdadm --grow /dev/md1 -n 2
```
Before doing the next step you must wait for the RAID syncing to finish (otherwise grub will not find what it is looking for). Wait for it to finish by running:
```
watch cat /proc/mdstat
```
Record information about the new meta-device so that mdadm can do certain operations without scanning the disks with the command:
```
{
    echo -e "DEVICE partitions\nMAILADDR root"
    mdadm --examine --scan | sed 's/$/ spares=0/'
} > /etc/mdadm/mdadm.conf
```
(If this step is done before the sync completes, then spurious ‘spares’ lines will appear in the config file.)
After the grub-install command above, the grub-loader will not be able to access /dev/md0 in order to load the kernel and we have just destroyed the old /dev/sda1 so booting with the above method will no longer work. However, grub-pc’s post-install script works, so now correct sda’s MBR by running:
```
dpkg-reconfigure grub-pc
```
(This will prompt for the list of devices onto which to write the GRUB MBR; select /dev/sda and /dev/sdb.)

Testing

Reboot the system again and verify that all MD devices are mounted.
Shut down the system and disconnect the first disk, boot up again.
Shut down the system again, reconnect the first disk, disconnect the second disk, boot up again.
Shut down the system again, reconnect the second disk, boot up again.
At this point the RAIDs will be desynchronised because each has been part of md0 without the other. To merge them back together do the following:
1. Determine which partition is no longer listed as a sub-device of /dev/md0 by running:
```
grep md0 /proc/mdstat
```
2. Re-add the device with:
```
mdadm /dev/md0 --add <missing-device>
```
  E.g. if the grep produced the following output then you must run following command to add the missing device:
```
torchio# grep md0 /proc/mdstat
md0 : active raid1 sdb1[0]
torchio# mdadm /dev/md0 --add /dev/sda1
mdadm: re-added /dev/sda1
torchio#
```
  At that point a resync will start
3. Determine which partition is no longer listed as a sub-device of /dev/md1 by running:
```
grep md1 /proc/mdstat
```
4. If the system didn’t swap then /dev/md1 will not have been pushed out of read-only mode and therefore the systems knows that they can be safely merged and has already done so. If not then you must run:
```
mdadm /dev/md1 --add <missing-device>
```
  At that point a resync will start.
5. You can monitor it with:
```
watch cat /proc/mdstat
```

Raiding remaining space

Note that some space will be wasted when a RAID5 device is set up across a set of disks which includes the disk with the OS and swap on it; we live with this.

Allocate remaining space on the first and second disks to single partitions using parted.
Insert any other disks.
Partition the other disks as per the first and second disk, but set each disk’s partition types as follows:
Attention: The internal data of table “100” is corrupted!

Set some environment variables:

DISKS=(<dev#1> <dev#2> ...)     #  E.g. DISKS=(/dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3)
RAID_LEVEL=<raid-level>         #  E.g. RAID_LEVEL=5

Create the array by running:

mdadm --create /dev/md2 --level=$RAID_LEVEL \
    --raid-devices=${#DISKS[*]} ${DISKS[*]}

Record information about the new meta-device so that mdadm can do certain operations without scanning the disks with the command:
```
{
    echo -e "DEVICE partitions\nMAILADDR root"
    mdadm --examine --scan | sed 's/$/ spares=0/'
} > /etc/mdadm/mdadm.conf
```

Allocate this new metadevice for LVM and create a volume group on it:

#  This will trigger the RAID synchronisation which was PENDING
pvcreate /dev/md2
vgcreate vg0 /dev/md2

Monitor the RAID synchronisation by running:
```
watch cat /proc/mdstat
```
and wait for the syncing to finish. You can also verify that the synchronisation is reading from and writing to the correct disks with:
```
iostat -p 1
```
Create a top-level mountpoint for all volumes in this volume group:
```
mkdir -p /vol
```

If the machine is not using NIS or is a NIS master server:

If the machine is running lenny (whose samba package suffers from bug BTS#502801) then run:

UID_MIN=$(sed -n 's/^UID_MIN[\t ]*//p' /etc/login.defs)
UID_MAX=$(sed -n 's/^UID_MAX[\t ]*//p' /etc/login.defs)
getent passwd | sort -u | awk -F: \
    "{ if ( \$3 >= $UID_MIN && \$3 <= $UID_MAX ) { print } }" | \
    /usr/sbin/mksmbpasswd > /etc/samba/smbpasswd
pdbedit -i smbpasswd -e tdbsam
rm /etc/samba/smbpasswd

If this storage server is serving home directories then edit /etc/samba/smb.conf and adjust the following settings:
```
[homes]
    ...
    read only = no
```
Run:
```
service samba restart
```

Creating and configuring shares

You can share three different things:

a new logical volume with a filesystem on it
a new logical volume without a filesystem on it
raw space inside a new large file (you might want to create a new logical volume with a filesytem on it to act as a container for such large files)

You can share using three different protocols:

NFS (only applicable to (a) above)
CIFS (only applicable to (a) above)
iSCSI (only applicable to (b) and (c) above)

All these cases are covered by the following procedure:

optionally create a logical volume to act as a container for large files
create a logical volume for sharing or create a large file in the container for sharing
optionally put a filesystem on it
configure NFS and/or CIFS access to the filesystem in the logical volume or configure iSCSI access to the raw space in the logical volume or large file

These steps are explained in detail below.

To create a logical volume as a container for large files:

Set some environment variables:

LVNAME=<name-of-container>           #  E.g. LVNAME=vmstore-$(uname -n)
VGNAME=vg0
LVSIZE=<size-in-GB>                  #  E.g. LVSIZE=150

Create, format and mount the container by running:

lvcreate --size ${LVSIZE}G --name $LVNAME /dev/$VGNAME
mkfs -t ext3 /dev/$VGNAME/$LVNAME
echo "/dev/$VGNAME/$LVNAME /vol/$LVNAME ext3 defaults 0 0" >> /etc/fstab
mkdir -p /vol/$LVNAME
mount /vol/$LVNAME

To create a logical volume for sharing:

Set some environment variables:

LVNAME=<name-of-share>               #  E.g. LVNAME=movies
VGNAME=vg0
LVSIZE=<size-in-GB>                  #  E.g. LVSIZE=1000

Create, format and mount the container by running:

lvcreate --size ${LVSIZE}G --name $LVNAME /dev/$VGNAME

To create a large file in the container for sharing:

Set some environment variables:

IMGNAME=<name-of-share>             #  E.g. IMGNAME=vm01-root-disk.img
IMGSIZE=<size-in-GB>                #  E.g. IMGSIZE=12
CONTAINERNAME=<name-of-container>   #  E.g. CONTAINERNAME=vmstore-$(uname -n)

Create the large file by running:

dd if=/dev/zero of=/vol/$CONTAINERNAME/$IMGNAME bs=1G count=$IMGSIZE

To create a filesystem on a logical volume:

Set some environment variables:

LVNAME=<name-of-share>              #  E.g. LVNAME=movies
VGNAME=vg0

Create a filesystenm and mount it:

mkfs -t ext3 /dev/$VGNAME/$LVNAME
echo "/dev/$VGNAME/$LVNAME /vol/$LVNAME ext3 defaults 0 0" >> /etc/fstab
mkdir -p /vol/$LVNAME
mount /vol/$LVNAME

To configure NFS access to the filesystem in the logical volume:
1. Add a suitable entry to /etc/exports.
2. Run:
```
exportfs -av
```
3. You may also want to add a suitable entry to the NIS server’s /etc/auto.staging file (not forgetting to remake the auto.staging NIS map afterwards). This entry might include something like:
```
... -noquota,noatime,nodiratime ...
```
To configure CIFS access to the filesystem in the logical volume:
1. Add a suitable entry to /etc/smb.conf. E.g. for /pub this might be as simple as:
```
[pub]
    comment = Public Archive
    browsable = yes
    path = /pub/
```
2. Run:
```
/etc/init.d/samba restart
```

To configure iSCSI access to the raw space in the logical volume or large file:

Set some environment variables:

CLIENT_HOSTNAME=<client-hostname>        #  E.g. CLIENT_HOSTNAME=ravioli
CLIENT_DISKNAME=<client-diskname>        #  E.g. CLIENT_DISKNAME=xvda

For access to a logical volume set some more environment variables:

VGNAME=vg0
LVNAME=<name-of-share>                   #  E.g. LVNAME=movies
TARGET_PATH=/dev/$VGNAME/$LVNAME
IOMODE=blockio

For access to a large file set some more environment variables:

IMGNAME=<name-of-share>                  #  E.g. IMGNAME=vm01-root-disk.img
CONTAINERNAME=<name-of-container>        #  E.g. CONTAINERNAME=vmstore-$(uname -n)
TARGET_PATH=/vol/$CONTAINERNAME/$IMGNAME
IOMODE=fileio

Derive the IQN for the iSCSI target by running:
```
IQN="iqn.$(date +%Y-%m).$(hostname -f | sed 's/\./\n/g' | tac | paste -d. -s):storage.disk.$CLIENT_HOSTNAME.$CLIENT_DISKNAME"
```
(Client’s also use IQNs but use the one specified in /etc/iscsi/initiatorname.iscsi; perhaps client IQN setup should be done in MDI and then server per-device IQN setup could also refer to the same file for the component of the device-specific IQN?)

Add details of the target to the server by running:

#  Have the target created at boot time
echo -e "Target $IQN\n\tLun 0 Path=$TARGET_PATH,Type=$IOMODE" >> /etc/ietd.conf
#  Have the target created right now
TID=$({ sed -n 's/tid:\([^ ]*\).*/\1/p' /proc/net/iet/volume; echo 0; } | sort -un | tail -1 | xargs expr 1 +)
ietadm --op new --tid=$TID --params Name=$IQN
ietadm --op new --tid=$TID --lun=0 --params Type=$IOMODE,Path=$TARGET_PATH

Verify the target is registered with:
```
cat /proc/net/iet/volume
```

Test it locally with:

iscsiadm --mode discovery --type sendtargets --portal localhost
iscsiadm --mode node --portal <portal-name> --targetname <target-name> --login
fdisk -l                                 #  verify disk now visible
iscsiadm --mode node --portal <portal-name> --targetname <target-name> --logout
fdisk -l                                 #  verify disk no longer listed

Backing up volumes

If the volume is shared without a filesystem, then back it up from within the VM or application that uses that raw space.

If the volume is shared with a filesystem, then consider and implement a backup policy; this may be as simple as running:

echo "17 1 * * * root rdiff-backup --exclude-other-filesystems \
    --exclude-sockets /vol/misc/shares/    /staging/backups/shared/" \
    >> /etc/cron.d/backup

Increasing the size of an NFS/CIFS volume

On NFS clients, stop any services accessing the volume and then unmount the volume.

Set some variables:

SIZEINC=<size-increase>        #  E.g. SIZEEINC=50G
DEVICE=<device>                #  E.g. DEVICE=/dev/vg0/lvol1
MNTPNT=<mountpoint>            #  E.g. MNTPNT=/vol/vmstore-macaroni

If the volume is NFS-exported on the NFS server then unexport it to allow it to be unmounted:
```
exportfs -u *:$MNTPNT
```
Unmount the volume:
```
umount $MNTPNT
```
Increase the size with:
```
lvextend -L +$SIZEINC $DEVICE
```
Resize the filesystem (this requires a filesystem check first):
```
e2fsck -f $DEVICE
resize2fs $DEVICE
```
Remount the volume:
```
mount $MNTPNT
```
Reexport the volume:
```
exportfs -av
```
On NFS clients remount the volume and then restart any services that were stopped earlier.

Storage log

Attention: The internal data of table “101” is corrupted!

Adding a second leg to one-legged DRBD devices

Run:

#  E.g. OTHERNODE=fiori
OTHERNODE=<name-of-other-node>
#  or whatever exists on $OTHERNODE
VMS="farfalle  fettuce  gnocchi  lasagne  penne  rocky  ziti"

Create the volumes needed:

for VM in $VMS; do lvcreate --name=$VM --size=12g vg0; done

Make them into DRBD devices:

for VM in $VMS; do scp $OTHERNODE:/etc/drbd/$VM.res /etc/drbd.d/; done
for VM in $VMS; do drbdadm create-md $X; done
for VM in $VMS; do drbdadm up $X; done

$OTHERNODE will have lost its connection so needs to be reconnected:
```
for VM in $VMS; do ssh -n $OTHERNODE drbdadm connect $VM; done
```

Verify that synchronisation has started, as in this example:

torchio# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
srcversion: EE47D8BF18AC166BE219757

 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:24704 dw:24704 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12557792
        [>....................] sync'ed:  0.3% (12260/12284)M
        finish: 8:43:14 speed: 272 (240) K/sec
 2: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:24704 dw:24704 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12557792
        [>....................] sync'ed:  0.3% (12260/12284)M
        finish: 8:43:14 speed: 272 (240) K/sec
 3: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:24936 dw:24936 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12557612
        [>....................] sync'ed:  0.3% (12260/12284)M
        finish: 8:43:14 speed: 320 (240) K/sec
 4: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:24704 dw:24704 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12557792
        [>....................] sync'ed:  0.3% (12260/12284)M
        finish: 8:43:14 speed: 284 (240) K/sec
 5: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:30440 dw:30440 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12554948
        [>....................] sync'ed:  0.3% (12260/12284)M
        finish: 8:43:07 speed: 320 (268) K/sec
 6: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:24708 dw:24708 dr:0 al:0 bm:1 lo:0 pe:6 ua:0 ap:0 ep:1 wo:b oos:12557788
        [>....................] sync'ed:  0.3% (12260/12284)M
        finish: 8:43:14 speed: 272 (240) K/sec
 7: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:24716 dw:24716 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:12557784
        [>....................] sync'ed:  0.3% (12260/12284)M
        finish: 8:43:14 speed: 272 (240) K/sec

torchio#

Verify the direction of the read and write using iostat on each node, as in this example:

torchio# iostat -x                          #  note high w/s and wsec/s
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   321.65    0.00  142.27     0.00  3546.39    24.93     2.04   14.38   5.83  82.89
sdb               0.00   332.99    0.00  130.93     0.00  3546.39    27.09     2.80   21.42   7.53  98.56
md0               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md2               0.00     0.00    0.00  464.95     0.00  3579.38     7.70     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00   61.86     0.00   494.85     8.00     1.64   26.53   4.87  30.10
dm-1              0.00     0.00    0.00   62.89     0.00   503.09     8.00     1.52   24.26   4.66  29.28
dm-2              0.00     0.00    0.00   61.86     0.00   494.85     8.00     1.61   26.07   4.53  28.04
dm-3              0.00     0.00    0.00   64.95     0.00   519.59     8.00     1.58   23.81   5.71  37.11
dm-4              0.00     0.00    0.00   84.54     0.00   626.80     7.41     1.09   13.02  12.00 101.44
dm-5              0.00     0.00    0.00   61.86     0.00   494.85     8.00     1.00   16.13   3.60  22.27
dm-6              0.00     0.00    0.00   61.86     0.00   494.85     8.00     2.05   33.20   6.07  37.53
torchio# ssh -n $OTHERNODE iostat -x        #  note high r/s and rsec/s
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sdb             170.00    49.00   34.00   20.00  1592.00   511.00    38.94     0.24    4.30   3.93  21.20
sda             174.00    51.00   35.00   18.00  1672.00   511.00    41.19     0.26    4.91   4.30  22.80
md0               0.00     0.00    0.00   16.00     0.00   128.00     8.00     0.00    0.00   0.00   0.00
md1               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md2               0.00     0.00  414.00   50.00  3312.00   377.00     7.95     0.00    0.00   0.00   0.00
dm-0              0.00     0.00   60.00    0.00   480.00     0.00     8.00     0.41    6.80   1.13   6.80
dm-1              0.00     0.00   60.00    0.00   480.00     0.00     8.00     0.24    4.00   0.67   4.00
dm-2              0.00     0.00   60.00    0.00   480.00     0.00     8.00     0.38    6.40   1.07   6.40
dm-3              0.00     0.00   60.00    0.00   480.00     0.00     8.00     0.29    4.80   0.80   4.80
dm-4              0.00     0.00   60.00    0.00   480.00     0.00     8.00     0.29    4.80   0.80   4.80
dm-5              0.00     0.00   60.00    0.00   480.00     0.00     8.00     0.22    3.60   0.60   3.60
dm-7              0.00     0.00   54.00   48.00   432.00   377.00     7.93     0.51    4.98   0.98  10.00
drbd7             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
drbd4             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
drbd2             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
drbd3             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
drbd6             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
drbd1             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
drbd5             0.00     0.00    0.00   47.00     0.00   376.00     8.00    39.88   74.64  21.28 100.00

Wait a long time!
Make the DRBD devices primary (so they are primary on both nodes):
```
for VM in $VMS; do drbdadm primary $VM; done
```

Introduction

Procedures

Raiding root and swap (lenny)

Raiding root and swap (squeeze)

Testing

Raiding remaining space

Creating and configuring shares

Backing up volumes

Increasing the size of an NFS/CIFS volume

Storage log

Adding a second leg to one-legged DRBD devices

See also