Configuring NAS services (revision 2.2)

Introduction

This procedure describes how setup network accessible storage services. It presupposes the availablilty of a system (physical or VM) with adequate storage space. It is intended for Debian 12.

NFS

  1. Run: 
    apt-get install nfs-kernel-server
  2. Create the filesytems to share (e.g. use lvcreate to create volumes and mkfs to make filesystem on them).
  3. Add entries to /etc/exports to export them.
  4. Run: 
    exportfs -av
  5. If the NFS server is a VM and on the same physical server there are also NFS clients then some performance increase can be gained by directing NFS traffic over a host-only network. To do this:
    1. On the physical server define a host-only network.
    2. On the NFS server and on the NFS clients add additional NICs connected to the network.
    3. Use the autmounter to specify that the client should first try to communicate over the host-only network and, after a timeout, fall back to the public network (for the case where one of the VMs gets migrated to a different server and has access to a different host-only network by the same name) by adding entries like this (from auto.staging): 
      svn -nordirplus,nfsvers=3,proto=tcp \
       filer-on-storage-net.pasta.net,\
       filer-on-public-net.pasta.net:/vol/small/svn
  6. Certain NFS mounts require special options that should be included in the client’s mount options (possibly via autofs maps):
    • access to a subversion repository requires nolock
    • if there is no need to update access times (not change times or modifcation times) then consider adding options noatime,nodiratime
  7. If the NFS server is not also an autofs client then, depending on what other services are being enabled in the storage server (e.g. SMB) then it may be necessary that users’ homes are accessible in the correct directory location (e.g. my home directory is /home/alexis but on the storage server this is actually /vol/home/alexis). If this is the case (and it is for me!) then create some symlinks to make things work. E.g.: 
    rmdir /home
ln -s vol/home /home
ln -s vol/pub /pub

SMB with LDAP authentication

This configuration uses Samba with a (possibly remote) LDAP backend for authentication. Note that this has not been testing with Debian 12 as I currently do not use SMB.

  1. On the Samba server:
    1. Run: 
      apt-get install samba
    2. Copy the file /usr/share/doc/samba/examples/LDAP/samba.ldif.gz to the LDAP server and unzip it into /tmp/samba.ldif.
    3. We will come back to configuring Samba shortly, but for now we just needed that file from the samba package copied to the LDAP server.
  2. On the LDAP server:
    1. Load the Samba schema and verify it was loaded by running: 
      ldapadd -Y EXTERNAL -H ldapi:/// -f /tmp/samba.ldif 
ldapsearch -LLLQY EXTERNAL -H ldapi:/// -b cn=schema,cn=config "(objectClass=olcSchemaConfig)" dn
    2. Allow users to change not only their userPassword attribute but also a couple of Samba-password-related attributes as follows:
      1. Locate the database entry (‘DN’) and the attribute definition that specify who can do what to what by running: 
        ldapsearch -W -Y EXTERNAL -H ldapi:/// -z 0     -b "cn=config"     "(objectclass=*)" | grep -B 10 -A 10 'attrs=userPassword'

        (Probably the entry is olcDatabase={1}mdb,cn=config.)

      2. Create /tmp/samba-pw-perms.ldif containing: 
        dn: <whatever-the-dn-was>
changetype: modify
replace: olcAccess
<all-olcAccess-lines-as-outputted-by-ldapsearch-above-yes-all-of-them>

        and then replace attrs=userPassword with attrs=userPassword,sambaNTPassword,sambaLMPassword.

      3. Run: 
        ldapmodify -H ldapi:// -Y EXTERNAL -f /tmp/samba-pw-perms.ldif
  3. Back on the Samba server:
    1. Install and configure smbldap-tools as follows:
      1. Run: 
        apt-get install smbldap-tools
      2. Run: 
        zcat /usr/share/doc/smbldap-tools/examples/smbldap.conf.gz \
    > /etc/smbldap-tools/smbldap.conf
cp /usr/share/doc/smbldap-tools/examples/smbldap_bind.conf \
    /etc/smbldap-tools/smbldap_bind.conf
      3. Edit /etc/smbldap-tools/smbldap.conf and set: 
        SID="<output-of-net-getlocalsid>"
sambaDomain="<single-upperase-word-description-of-network>"
slaveLDAP="ldap://ziti.pasta.net/"
masterLDAP="ldap://ziti.pasta.net/"
ldapTLS="0"
suffix="dc=pasta,dc=net"
usersdn="ou=Users,${suffix}"
computersdn="ou=Computers,${suffix}"
idmapdn="ou=Users,${suffix}"
      4. Run: 
        chmod 0644 /etc/smbldap-tools/smbldap.conf
chmod 0600 /etc/smbldap-tools/smbldap_bind.conf
    2. The default /etc/samba/smbd.conf is a bit unreadable. Clean it by running: 
      cd /etc/samba
testparam -s > smb.conf.new
systemctl stop smbd
mv smb.conf  smb.conf.orig
mv smb.conf.new smb.conf
systemctl start smbd
    3. Edit /etc/samba/smbd.conf and set/comment-out: 
      [global]
    passdb backend = ldapsam:ldap://ziti.pasta.net/
    ldap admin dn = cn=admin,dc=pasta,dc=net
    ldap suffix = dc=pasta,dc=net
    # Next to lines depend on TLS/SSL availability on LDAP server
    #ldap ssl = start tls
    ldap ssl = no
    ldap passwd sync = yes
    obey pam restrictions = no
    ldap user suffix = ou=Users
    ldap group suffix = ou=Groups
    ldap machine suffix = ou=Computers
    ldap idmap suffix = ou=Users
    #  Don't use Samba's internal LDAP syncer
    ldap passwd sync = no
    #  Use an external program
    unix password sync = Yes
    passwd program = /usr/sbin/smbldap-passwd -u %u
    passwd chat = *New*password* %n\n *Retype*new*password* %n\n *all*authentication*tokens*updated*
    #  'hide dot files' just sets dot files to be show up with the 'H'
    #  attribute; if you really want to hide them (from smbclient)
    veto files = .*
    4. If you want to do user creation from Windows then see /usr/share/doc/smbldap-tools/README.Debian.gz for details.
    5. Regarding resources, edit /etc/samba/smbd.conf and set: 
      ...

[homes]
...
read only = no
 ...

[pub]
    comment = Public Archive
    browsable = yes
    path = /pub/

#[printers]
#...

#[print$]
#...

      and restart smbd.

    6. Create the necessary structures in the LDAP database by running this (still on the Samba server): 
      smbldap-populate
    7. For each existing user in LDAP that needs SMB access run: 
      smbpasswd -a <username>

      (Note that this will change the user’s Unix password too!)

  4. To test go to a machine with smbclient installed run the following as a normal use: 
    lagane$ smbclient '\\stortini\alexis'
mkdir failed on directory /var/run/samba: Permission denied
Unable to initialize messaging context
Enter WORKGROUP\alexis's password: 
Try "help" to get a list of possible commands.
smb: \> ls
...
smb: \>
  5. Regarding the error messages:
    • mkdir .... Permission denied seems to be cosmetic (if I tell smbclient to contact my old non-LDAP-ised Samba server then it produces the same error message but nonetheless successfully connects to my home directory).
    • Unable to ... is due to a bug in smbclient but may also only be cosmetic; output in BTS#931688 seems to indicate that the smbclient shell can be started even with this message.

    HTTP/S

    In an earlier incarnation, my front-end webserver was an NFS client and could automount /pub and /home and could therefore server content from the NAS itself. However, the front-end webserver has become an LXC container, which means it cannot be an NFS client. For this reason we now enable HTTP/S access directly on the NAS.

    1. Ensure usernames are resolvable (this is needed so that http://hostname/~username can be expanded): 
      getent -s ldap alexis
id alexis
ls ~alexis/

      Possible causes of the first command working but the later ones not are:

      1. nscd needs restarting
      2. nscd needs stopping and disabling altogther
      3. /etc/nsswitch.conf wrong
    2. Run: 
      apt -y install apache2
    3. Choose whether to use http or https on the backend webserver (I choose http).
    4. By default only the http site 000-default is enabled: 
      stortini# a2query -s
000-default (enabled by site administrator)
stortini#

      so if the https site is wanted (which for me it is) then you need to run:

      a2dissite 000-default
a2ensite default-ssl
a2enmod ssl
vi /etc/apache2/ports.conf  #  Comment out 'Listen 80'
systemctl restart apache2
    5. Edit /etc/apache2/sites-enabled/*.conf and add: 
      #  Do not mix 'Location' and 'Directory'; it is too complicated to work
#  out what will match! 
#
#  Of all the matching stanzas, the last one wins!
<LocationMatch ^(?!/~(?:<list-of-pipe-separated-users>)/).*$>
    Require all denied
    Options None
    ErrorDocument 403 Forbidden.
</LocationMatch>
    6. Note the comments added to the config file in the previous step, which mean that each resource we now want to allow access to will be specified (1) using Location and not Directory, (2) after the the above stanza!
    7. To allow access to personal web pages:
      1. Edit /etc/apache2/mods-available/userdir.conf and change all occurences of ‘public_html’ to ‘.public_html’
      2. Edit /etc/apache2/mods-available/userdir.conf and change any occurrences of ‘SymLinksIfOwnerMatch’ to ‘FollowSymLinks’
      3. Run: 
        a2enmod userdir
    8. To allow access to fancy indexes follow the procedure described at Configuring web services (revision 2.1).
    9. Add stanzas for the local public and private Debian package repositories: 
      Alias /debian/ "/pub/computing/software/local/debian/localpublic-deb/"
<Location /debian/>
    Require all granted
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
</Location>

Alias /debian-private/ "/pub/computing/software/local/debian/localprivate-deb/"
<Location /debian-private/>
    Require ip 192.168.0.0/16
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
</Location>
    10. Add similar stanzas for the local public and private Redhat package repositories.
    11. Add similar stanzas for the local public and private source repositories
    12. Restart the website with: 
      systemctl restart apache2
    13. Configure the front-end webserver according to Configuring web services (revision 2.1); specifically the sections on proxying to a backend webserver for sites home.pasta.freemyip.com and repo.pasta.freemyip.com.

    DLNA

    1. Install the server as follows: 
      apt -y install minidlna
systemctl stop minidlna
    2. Edit /etc/minidlna.conf and set: 
      media_dir=A,/var/lib/minidlna/audio
media_dir=V,/var/lib/minidlna/video
merge_media_dirs=yes
friendly_name=<whatever-you-prefer>
serial=1
wide_links=yes
    3. Create a symlink farm under /var/lib/minidlna to expose only certain parts of your media collection. For example: 
      cd /var/lib/minidlna
mkdir video audio
ln -s /pub/multimedia/video/{cinema,music,television,trailers} video/
ln -s /pub/multimedia/audio/{music,spoken/{comedy,drama}} audio/
    4. Wipe the cache, restart minidlna and monitor its CPU usage (which should be high for some length of time, depending how much media there is to index): 
      rm /var/cache/minidlna/files.db
systemctl start minidlna
top

    Data migration

    1. Rsync data across from the old server.
    2. Test that NFS/SMB clients can mount and write the new server.
    3. Regarding LDAP:
      1. On the LDAP server make sure shelldap is installed and configured; i.e. ~/.shelldap.rc is set up with something like: 
        server: localhost
binddn: cn=admin,dc=pasta,dc=net
bindpass: <admin-password>
basedn: dc=pasta,dc=net
tls: no
      2. Use shelldap to verify that all automounter maps use a DNS CNAME (e.g. nas1.example.com) rather than an A nane (e.g. pluto.example.com).
      3. If some shares are not being migrated yet (e.g. I did not migrate svn) then the automounter entries for these need to updated to use the A name or IP address!
    4. Shut down or block access to the following services, which might update data on the old server:
      • cronjobs (e.g. get_iplayer)
      • log out as many users as possible
    5. Rsync data across from the old server again (don’t forget --delete option this time).
    6. Update DNS to make the CNAME point to the new server, for me this means updating:
      1. filer-on-storage-net.pasta.net IN CNAME …
      2. filer-on-public-net.pasta.net IN CNAME …
      3. On all hosts run: 
        nscd -i hosts
    7. Block access to the data on the old server but do it in a reboot-safe way (it’s probably a good idea to reboot the old server too)
    8. Disable backup of the old server (as the previous step may remove backups)
    9. Resolve any hanging mounts.
    10. To save doing a slow full backup on the backup server, move or clone the just-migrated data directories to the new host.

    See also