Introduction
This page describes various procedures related to LVM.
Procedures
Closing open LVs
LVs must be deactivated before removal, although lvremove can do both at the same time, but only if the volume is already closed. “Closed” means not mounted, not held open by an application (typically it is database applications which hold open storage devices without filesystems). But sometimes it can be a little more complicated.
- Verify that the LV really is open by running:
lvdisplay /dev/<vgname>/<lvname>
(‘1’ means it is open, ‘0’ means it is closed), as in this example:
spaghetti# lvdisplay /dev/vg0/pubvol | grep open # open 1 spaghetti#
- Try each of the following steps and then re-check if the LV is still open; if it is then there is no need to continue to the next step.
- Refresh the cache of LV statuses by running:
lvscan
- If the LV has a filesystem on it, verify that the LV is not mounted by running the following command and reviewing the output:
mount
- Verify no process is holding the device open by running:
lsof | grep <lvname>
- If the LV has partitions on it (as might be the case if the LV is presented as a disk to a VM) and you scanned the LV for partitions (using partprobe or kpart) then kernel will “acquainted” with these partitions and and that “acquaintment” keeps the container LV open. You can verify if this is the case by looking under /dev for names of devices containing the LV name, as in this example:
spaghetti# find /dev -ls | grep pubvol 121092188 0 lrwxrwxrwx 1 root root 38 Apr 14 09:06 /dev/vg0/pubvol -> /dev/mapper/vg0-pubvol 120827538 0 brw-rw---- 1 root disk Apr 13 16:55 /dev/mapper/vg0-pubvol 120828929 0 brw-rw---- 1 root disk Apr 13 16:04 /dev/mapper/vg0-pubvolp1 spaghetti#
Serverfault says that the kernel can be “unacquainted” with these partitions by running:
kpartx -dv /dev/<vgname>/<lvname>
- Since the above output shows that device mapper was involved in the management, I wondered if the device mapper was holding the container LV open. I ran:
dmsetup deps | grep <lvname>
and this showed that the device mapper understood that the partition depended on the container LV, as in this example:
spaghetti# dmsetup deps | grep pubvol vg0-pubvolp1: 1 dependencies : (253, 128) vg0-pubvol: 1 dependencies : (253, 2) spaghetti#
By looking in /dev/mapper for devices with listed minor numbers, it is possible to establish what depends on what. In the above example, “(253, 128)” means that vg0-pubvolp1 depends on vg0-pubvol, which could explain why vg0-pubvol is being held open. So then I just asked the device mapper to delete the the “child” in this dependency relation ship with:
dmsetup remove /dev/mapper/<vgname>-<lvname>p*
After that I reran dmsetup deps, which now showed:
spaghetti# dmsetup deps | grep copy vg0-pubvol: 1 dependencies : (253, 2)
(I.e. nothing depends upon this device any more.)
- Refresh the cache of LV statuses by running:
Creating, splitting and rejoining LVM RAID 10 volumes
This page describes how Alexis Huxley created a test RAID10 volume on LVM, how he split the mirror specifying which disks to remove from the volume, in order to apply firmware upgrades on the two removed disks, and how he re-added the disks afterwards.
- Allocate the PVs for LVM usage:
spaghetti# pvcreate /dev/xvdb Physical volume "/dev/xvdb" successfully created spaghetti# pvcreate /dev/xvdc Physical volume "/dev/xvdc" successfully created spaghetti# pvcreate /dev/xvdd Physical volume "/dev/xvdd" successfully created spaghetti# pvcreate /dev/xvde Physical volume "/dev/xvde" successfully created spaghetti#
- create the VG from them:
spaghetti# vgcreate vg0 /dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde Volume group "vg0" successfully created spaghetti#
- Create a RAID10 LV from that VG specifying PVs in order:
<leg0-part0> <leg0-part1> ... <leg1-part0> <leg1-part1> ...
which for my setup meant running:
spaghetti# lvcreate --mirrors 1 --name lvol0 --mirrorlog core --extents 510 vg0 \ /dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde Logical volume "lvol0" created spaghetti#(Note that I first tried to allocate size --size 1e which was impossibly big but told me the number of extents available, which I halved to get 510).
- Monitor things with iostat. At first iostat showed:
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvda 0.00 0.00 0.00 0 0 xvdb 198.48 12715.15 0.00 25176 0 xvdc 0.00 0.00 0.00 0 0 xvdd 198.99 0.00 12735.35 0 25216 xvde 0.00 0.00 0.00 0 0 dm-0 197.98 12670.71 0.00 25088 0 dm-1 197.98 0.00 12670.71 0 25088 dm-2 0.00 0.00 0.00 0 0
and a little bit later it showed:
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvda 0.00 0.00 0.00 0 0 xvdb 0.00 0.00 0.00 0 0 xvdc 93.47 5981.91 0.00 11904 0 xvdd 0.00 0.00 0.00 0 0 xvde 89.45 0.00 5724.62 0 11392 dm-0 89.45 5724.62 0.00 11392 0 dm-1 93.47 0.00 5981.91 0 11904 dm-2 0.00 0.00 0.00 0 0
and a little bit later still it showed:
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn xvda 0.00 0.00 0.00 0 0 xvdb 0.00 0.00 0.00 0 0 xvdc 0.00 0.00 0.00 0 0 xvdd 0.00 0.00 0.00 0 0 xvde 0.00 0.00 0.00 0 0 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0
During the first output above lvs --all showed:
spaghetti# lvs --all LV VG Attr LSize Origin Snap% Move Log Copy% Convert lvol0 vg0 mwi-a- 1.99g <less-than-50%> [lvol0_mimage_0] vg0 Iwi-ao 1.99g [lvol0_mimage_1] vg0 Iwi-ao 1.99g
and during the second output it showed:
spaghetti# lvs --all LV VG Attr LSize Origin Snap% Move Log Copy% Convert lvol0 vg0 mwi-a- 1.99g <more-than-50%> [lvol0_mimage_0] vg0 Iwi-ao 1.99g [lvol0_mimage_1] vg0 Iwi-ao 1.99g
- In a moment you’ll split the mirror, but to confirm that you’ve removed the devices you want to remove it is useful to run the following command now for comparison purposes later:
spaghetti# pvs PV VG Fmt Attr PSize PFree /dev/xvdb vg0 lvm2 a- 1020.00m 0 /dev/xvdc vg0 lvm2 a- 1020.00m 0 /dev/xvdd vg0 lvm2 a- 1020.00m 0 /dev/xvde vg0 lvm2 a- 1020.00m 0 spaghetti#
- Split mirror, specifying what to remove:
spaghetti# lvconvert --mirrors 0 /dev/vg0/lvol0 /dev/xvdb /dev/xvdc
- Confirm that you’ve removed the devices you wanted to remove run:
spaghetti# pvs PV VG Fmt Attr PSize PFree /dev/xvdb vg0 lvm2 a- 1020.00m 1020.00m /dev/xvdc vg0 lvm2 a- 1020.00m 1020.00m /dev/xvdd vg0 lvm2 a- 1020.00m 0 /dev/xvde vg0 lvm2 a- 1020.00m 0 spaghetti#
(Note that /dev/xvdb and /dev/xvdc are now back in the ‘free’ pool.)
- This is the moment to do whatever it is you have to do the the removed PVs (e.g. firmware upgrade on SAN).
- Join mirror, specifying what to add again:
spaghetti# lvconvert --mirrors 1 --mirrorlog core /dev/vg0/lvol0 /dev/xvdb /dev/xvdc
(Note that without the --mirrorlog option there are not enough extents available on the devices being added because the log will be allocated out of them.)