Configuring storage services generation one

Introduction

This page documents my experience building a ZFS Linux fileserver. The hostname allocated for the system was ‘spirali’.

Despite being extremely happy with the functionality of the products selected, I was not happy with spirali’s performance, in part due to the old hardware but mostly due to ZFS’s high CPU demands and it running in userspace via the FUSE module.

Eventually it was replaced by macaroni, but this page is kept as a record of how I installed it, in the hope that it might be of use to others.

This procedure has been superceded by Configuring storage services generation two; it remains here for reference.

Analysis and design

Requirements:

  1. free
  2. Unix-like
  3. RAID5
  4. snapshotting
  5. iSCSI target
  6. access protocols: NFS, SMB/CIFS
  7. directory lookup: NIS

Considered products:

  1. FreeNAS
  2. OpenFiler
  3. Linux+MD+LVM2
  4. Linux+BTRFS
  5. Linux+MD+rdiff-backup
  6. Belenix+ZFS
  7. Nexenta+ZFS
  8. OpenSolaris+ZFS
  9. Linux+FUSE+ZFS+kernel-nfs-server
  10. Linux+FUSE+ZFS+unfs3

Problematic products:

  1. Nexenta, Belenix: no driver for my SATA controller card
  2. BTRFS: not stable
  3. FreeNAS: easily reproducible crashes of rsyncs on ZFS, does not support NIS
  4. OpenSolaris: assume same driver issues as Nextenta and Belenix
  5. LVM2: snapshotting speed rapidly degrades
  6. rdiff-backup: storage requirements high
  7. OpenFiler: uses LVM2 for snapshotting
  8. FUSE+ZFS+kernel-nfs-server: as documented in /usr/share/doc/fuse/README.NFS, ESTALE errors with kernel NFS, this can be worked around by having clients mount with the ‘noac’ option, but, as documented in nfs(5) this triggers a massive performance hit
  9. FUSE+ZFS+unfs3: although this solves the ESTALE error, it causes programs to hang with waiting for NFS locks which unfs3 does not implement; having clients mount with the ‘nolock’ option solves this but when I tried to run programs stored in my home on NFS then I got exec() errors.
  10. ZFS+FUSE: slow!

Proposed configuration:

  1. disks: 4 x 1TB SATA
  2. RAM: 4GB
  3. OS: Debian
  4. filesystem: FUSE+ZFS
  5. pools: ZFS RAIDZ (4TB gross, 3TB net)
  6. datasets: pub, home, mail, wikis, svnrepos, logs (apache2), p2p (bittorrent, mldonkey, i2p, azureus), iscsi (for VMs) all with individual snapshotting frequencies and retention periods
  7. filerserver daemons: NFSv3 using kernel-nfs-server with ‘noac’ to prevent ESTALE errors on all clients, Samba, iSCSI target server
  8. other daemons: DHCP, no-ip, dynamic DNS, VMs, NTP, NIS
  9. snapshotting: 24 x hourly, 31 x daily
  10. desktop environment: none

Still to think about

  • try NILFS
  • try cachefs on NFS clients

Installation log

OS

Obsolete instructions removed.

ZFS
  1. Pre-packaged versions of FUSE don’t support NFS, so compile and install FUSE from sources as follows: 
    #  Divert the standard fuse modules
dpkg-divert --divert /lib/modules/`uname -r`/kernel/fs/fuse/fuse.ko.no-nfs --rename /lib/modules/`uname -r`/kernel/fs/fuse/fuse.ko
#  install fuse-utils so that can divert a file later provided by
#  fuse sources but that would be installed later as a zfs-fuse dependency.
apt-get install fuse-utils
dpkg-divert --divert /etc/init.d/fuse.no-nfs --rename /etc/init.d/fuse
#  download, compile, install new fuse
wget ???/fuse-2.7.4.tar.gz
tar xzf fuse-2.7.4.tar.gz
cd fuse-2.7.4
apt-get install gawk make gcc linux-headers-2.6-686
./configure --enable-kernel-module
make
make install
  2. Install ZFS modules by running: 
    apt-get install devscripts build-essential zlib1g-dev libfuse-dev scons debhelper dpatch xsltproc docbook-xsl
dget http://www.fushizen.net/zfs-fuse/zfs-fuse_0.4.0~beta1.hg20070508-1.dsc
dpkg-source -x zfs-fuse_0.4.0~beta1.hg20070508-1.dsc
cd zfs-fuse-0.4.0~beta1.hg20070508
dpkg-buildpackage -us -uc -b
  3. Optionally archive the resulting .deb file.
  4. Install the following packages and their dependencies:
    • zfs-fuse
  5. Set up snapshotting using Alexis Huxley‘s script by running: 
    mkdir -p /usr/local/opt
svn co https://svn.pasta.freemyip.com/main/storagetools/trunk /usr/local/opt/storagetools
#  Only do this next line if AhTools is installed.
package-symlink /usr/local/opt/storagetools /usr/local/bin  /usr/local/doc/storagepools
touch /etc/snapshot.conf
#  Adjust path accordingly if AhTools not installed.
echo "*  * * * * root /usr/local/bin/snapshot -d 100 process     >> /pool0/logs/snapshot/snapshot.log 2>&1" > /etc/cron.d/zfs-snapshot
echo "27 0 * * 0 root /usr/local/bin/snapshot -d 100 scrub pool0 >> /pool0/logs/snapshot/scrub.log    2>&1" > /etc/cron.d/zfs-scrub
  6. To configure the ZFS pools complete the following sub-procedure:
    1. Set some variables: 
      POOLNAME=<poolname>                   #  E.g. POOLNAME=pool0
DISKS="<list-of-disks-to-use>"        #  E.g. DISKS="sda sdb sdc sdd"
    2. Run: 
      zpool create -m none $POOLNAME raidz $DISKS

      (The -m none means don’t mount the pool.)

  7. To configure the ZFS datasets complete the following sub-procedure:
    1. Set some more variables: 
      DATASETNAMES=<list-of-dataset-names>  #  E.g. DATASETNAMES="pub home mail wikis svnrepos logs bittorrent iscsi"
    2. Run: 
      for DATASETNAME in $DATASETNAMES; do
    zfs create -o mountpoint=/$POOLNAME/$DATASETNAME $POOLNAME/$DATASETNAME
done

      (The -o mountpoint=... means, unlike the pool, do mount the datasets.)

    3. Set up suitable snapshotting entries in /etc/snapshot.conf. E.g. 
      '[ `date +%M`     = 00     ]' pool0/home     hourly 24
'[ `date +%H%M`   = 0000   ]' pool0/home     daily  61
'[ `date +%M`     = 00     ]' pool0/mail     hourly 24
'[ `date +%H%M`   = 0000   ]' pool0/mail     daily  61
'[ `date +%H%M`   = 0000   ]' pool0/iscsi    daily  31
'[ `date +%H%M`   = 0000   ]' pool0/svnrepos daily  31
'[ `date +%H%M`   = 0000   ]' pool0/wikis    daily  31
'[ `date +%H%M`   = 0000   ]' pool0/logs     daily  31
'[ `date +%w%H%M` = 00000  ]' pool0/pub      weekly 8
  8. Since some of the ZFS datasets will be exported via NFS, the ZFS startup scripts must be moved to start before the NFS startup scripts. Do this by running: 
    #  fuse-zfs starts at 23, so fuse must start at 23
rm /etc/*.d/[SK]??fuse
update-rc.d fuse start 23 2 3 4 5 . stop 23 0 1 6 .
#  unfs3 starts at 25, so zfs-fuse must start at 24
rm /etc/*.d/[SK]??zfs-fuse
update-rc.d zfs-fuse start 24 2 3 4 5 . stop 24 0 1 6 .
iSCSI

ZFS on FUSE doesn’t provide iSCSI target support, but the kernel itself can provide this directly.

  1. Run the following commands: 
    #  Enable iSCSI target server
apt-get install iscsitarget-modules-2.6-686 iscsitarget
perl -pi -e 's/false/true/' /etc/default/iscsitarget
perl -pi -e 's/^/#/' /etc/iet.conf
/etc/init.d/iscsitarget start
#  Install tools for testing
apt-get install open-iscsi dosfstools
#  Define IQN needed when creating all iSCSI volumes
IQN_PREFIX=iqn.<yyyy>-<mm>-<fqhn-in-reverse>    #  E.g. IQN_PREFIX=iqn.2009-06-net.pasta.spirali
  2. For each required iSCSI volume complete the following sub-procedure:
    1. Run the following commands: 
      VOLNAME=<volume-name>                           #  E.g. VOLNAME=ravioli-sda
VOLSIZE=<size-in-MB>                            #  E.g. VOLSIZE=1024
IQN_SUFFIX=storage.disk<num>.sys<num>.<tag>     #  E.g. IQN_SUFFIX=storage.disk0.sys0.ravioli-sda
    2. Create a file to use as an iSCSI volume inside the dedicated ZFS dataset: 
      dd if=/dev/zero bs=1M count=$VOLSIZE of=/$POOLNAME/iscsi/$IQN_PREFIX:$IQN_SUFFIX
    3. Register the file as an iSCSI target by running: 
      echo -e "Target $IQN_PREFIX:$IQN_SUFFIX\n\tLun 0 Path=/$POOLNAME/iscsi/$IQN_PREFIX:$IQN_SUFFIX,Type=fileio" >> /etc/ietd.conf
/etc/init.d/iscsitarget restart
cat /proc/net/iet/volume
    4. Test it locally with: 
      iscsiadm --mode discovery --type sendtargets --portal localhost
iscsiadm --mode node --targetname <target-name> --portal <portal-name> --login
fdisk -l #  verify disk now visible
fdisk /dev/<scsi-device>     #  create a partition spanning whole disk of type 'b' (W95)
mkfs -t vfat /dev/<scsi-device>1
mount /dev/<scsi-device>1 /mnt
df
umount /mnt
iscsiadm --mode node --targetname <target-name> --portal <portal-name> --logout
fdisk -l #  verify disk no longer listed
  3. Since some of the ZFS datasets will be exported via iSCSI, the ZFS startup scripts must be moved to start before the NFS startup scripts, or the iSCSI startup scripts must be moved to start after the ZFS startup scripts. Do this by running: 
    rm /etc/*.d/[SK]??iscsitarget
update-rc.d iscsitarget start 25 2 3 4 5 . stop 25 0 1 6 .
NIS
  1. Configure as a NIS master server and a NIS client.
  2. Configure autofs.
  3. Populate the following NIS maps:
    1. group
    2. passwd
    3. auto.home
    4. auto.staging

    and then run:

    make -C /var/yp
DHCP & dynamic DNS
  1. configure name resolution.
NFS & SMB
  1. Run: 
    apt-get -y install nfs-kernel-server samba

    1.

  2. Run: 
    POOLNAME=<poolname>         #  E.g. POOLNAME=pool0
DATASETNAME=<dataset-name>  #  E.g. DATASETNAME=pub
EXPORTMASK=<cidr>           #  E.g. EXPORTMASK=192.168.1.0/24
SHARENAME=<smb-share-name>  #  E.g. SHARENAME=pub
#  Do the next two lines to share via NFS
echo "/$POOLNAME/$DATASETNAME $EXPORTMASK(rw,no_subtree_check,fsid=$RANDOM)" >> /etc/exports
exportfs -av
#  Do the next two lines to share via SMB
echo -e "[$SHARENAME]\n\tbrowseable = yes\n\tread only = no\n\tpath = /$POOLNAME/$DATASETNAME\n" >> /etc/samba/smb.conf
killall -HUP smbd
Web services
  1. Configure web services.
Transmission
  1. Configure transmission.

See also