Configuring mail services (revision 2)

Introduction

This page describes how Alexis Huxley installed and configured various mail services.

Warning: Postfix’s chroot environment

Postfix’s chroot environment contains copies of system files, which are not updated when using the postfix command to control the service. Therefore always using the systemctl command instead, otherwise a lot of head scratching ensues!

A self-signed certificate

This procedure will attempt to use a Let’s Encrypt SSL certificate, but that relies on the host having the internet-visible name. Since we have an old mail server in service and this host will replace it, the internet-vislble name has not yet been transferred to it. So we start with a self-signed certificate and switch later.

  1. Create a self-signed certificate by running: 
    openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -keyout /etc/ssl/private/mail.key -out /etc/ssl/certs/mail.pem

    and answer the questions. E.g.:

    Country Name (2 letter code) [AU]:DE
State or Province Name (full name) [Some-State]:Bayern
Locality Name (eg, city) []:Ismaning
Organization Name (eg, company) [Internet Widgits Pty Ltd]:None
Organizational Unit Name (eg, section) []:  Common Name (e.g. server FQDN or YOUR name) []:
    <fqdn-mail-clients-use-to-connect-to-mail-server>
Email Address []:<my-personal-email-address>

    If the common name does not match the name that clients use to connect to the mail server then mutt will complain:

    WARNING: Server hostname does not match certificate
  2. Make sure the key is readable only by root by running something like this: 
    root# ls -ld /etc/ssl/private
drwx--x--- 2 root ssl-cert 4096 Nov 2 14:39 /etc/ssl/private
  3. If you have followed this procedure in order to regenerate a certificate then restart the dovecot service.

Dovecot

We set up Dovecot first because it provides user information to Postfix.

  1. Before running the next command, note that it will probably fail due to IPv6 being disabled. Don’t panic.
  2. Run: 
    apt-get install dovecot-core dovecot-imapd dovecot-ldap
  3. Stop dovecot with: 
    systemctl stop dovecot
  4. Entirely replace /etc/dovecot/dovecot.conf with the following: 
    #  Don't listen on IPv6
listen = *

#  Encryption
ssl = required
ssl_cert = </etc/ssl/certs/mail.pem
ssl_key = </etc/ssl/private/mail.key
disable_plaintext_auth = no

#  Services provided to other programs
service auth {
    unix_listener /var/spool/postfix/private/auth {
        group = postfix
        mode = 0660
        user = postfix
    }
}

#  Protocols 
protocols = " imap"

#  Misc
mail_privileged_group = mail
mail_location = maildir:/var/mail/maildir/%u:INDEX=/var/mail/indexes/%u
  5. If you want to store mail on NFS but indexes on local disk in an environment where only one machine will access the NFS mount (I don’t!):
    1. Add the following to /etc/dovecot/dovecot.conf: 
      #  single-server NFS setup
mmap_disable = yes
mail_fsync = always
mail_nfs_storage = no
mail_nfs_index = no
    2. NFS mount the /var/mail/maildir. For me, this meant:
      1. on the storage server, add an entry to /etc/exports and run exportfs -av
      2. on the LDAP server, add a suitable entry to the auto.staging map (see here for details)
      3. on the mail server, replace /var/mail/maildir with a symlink to ../../staging/mail/maildir.
  6. If you want to have Dovecot automatically add a Trash and Sent folder to mailboxes (I don’t!) then add the following to /etc/dovecot/dovecot.conf: 
    protocol imap {
    mail_plugins = " autocreate"
}
plugin {
    autocreate = Trash
    autosubscribe = Trash
    autocreate2 = Sent
    autosubscribe2 = Sent
}
  7. If you want to have Dovecot authenticate users and for Dovecot’s LDA to get their home directories from /etc/passwd or from NIS (I did but not any longer) then:
    1. Add the following to /etc/dovecot/dovecot.conf: 
      #  User and home directory lookup from files/NIS
userdb {
    driver = passwd
}

#  Authenticate via /etc/pam.d/dovecot, which will read files/NIS
passdb {
  driver = pam
  args = dovecot
}
  8. If you want to have Dovecot authenticate users and for Dovecot’s LDA to get their home directories from LDAP (I do!) then:
    1. Add the following to /etc/dovecot/dovecot.conf: 
      #  User and home directory lookup from LDAP
userdb {
    driver = ldap
    args = /etc/dovecot/dovecot-ldap.conf
}

#  Authenticate via LDAP
passdb {
  driver = ldap
  args = /etc/dovecot/dovecot-ldap.conf
}
    2. Create /etc/dovecot/dovecot-ldap.conf containing: 
      #  Without these two, dovecot will log messages complaining
#  explicitly about these two
base = ou=Users,dc=pasta,dc=net
uris = ldaps://ldap.pasta.freemyip.com/

#  Without these two, dovecot syslogs "dovecot: imap-login: 
#  Aborted login (auth failed, ...): user=, method=PLAIN, 
#  rip=..., lip=..., TLS, session="
auth_bind = yes
auth_bind_userdn = uid=%u,ou=Users,dc=pasta,dc=net

#  Without these two (one commented out and one not!), dovecot
#  will read TLS_CACERT from /etc/ldap/ldap.conf
#  and if that doesn't exist then dovecot will fail to authenticate
#  users (and postfix will refuse too as it delegates authentication
#  to dovecot).
#tls = yes
tls_ca_cert_file = /etc/ssl/certs/ca-certificates.crt

#  If you want to set any additional user-specific attributes
#  (e.g. mail quota) then set ALL NEEDED user-specific 
#  attributes plus the additional ones.
user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid
  9. If the mail directories are to be automounted, then make sure there is a suitable entry in the automounter map; see here for details.
  10. It is probable that the indexes and maildir directories do not exist under /var/mail. Create them  and correct permissions on /var/mail that may be caused by it being a new disk by running: 
    cd /var/mail
chown root:mail .
chmod 2775 .
mkdir indexes maildir
chown root:mail indexes maildir
chmod 3777 indexes maildir
  11. Restart stuff: 
    service dovecot restart
  12. Testing:
    1. Connect to your usual mailserver (not the one we’re setting up now) save a copy of a mail to a new mail folder (i.e. a file) in your home directory.
    2. Connect to the new mail server, change folder to the mail folder you just created and save that mail back to your inbox on the new server.
    3. Change folder to your inbox and verify the just saved mail is there and accessible.

Postfix

  1. Run: 
    apt-get -y  install postfix postgrey
service postfix stop
  2. Set some environment variables that will be used later in this procedure; you should adjust the values to match your own requirements: 
    MY_FQHN=$(hostname -f)
MY_ROOTRCPT=alexis@pasta.net
MY_OTHERNAMES="mail.pasta.freemyip.com,pasta.freemyip.com"   #  Note comma-separated
RELAY_HOSTNAME=smtp.gmail.com
RELAY_AUTHENTICATE=yes
RELAY_LOGIN=alexishuxley
RELAY_PASSWD=<my-gmail-password>
  3. Edit /etc/postfix/main.cf and insert the following basic configuration: 
    ######################################################################
#
#  WHO AM I AND WHAT INTERFACES DO I LISTEN ON?
#
######################################################################

#  myhostname is mainly used to derive other stuff
myhostname = MY_FQHN
#  myhostname must be set and cannot be derived from $myorigin. On 
#  the other hand, myorigin defaults to $myhostname. Therefore set 
#  myhostname explicitly and leave myorigin unset. mydomain defaults
#  to the last parts of $myhostname.
#myorigin = ...
#mydomain = ...
#  Which interfaces to listen on for incoming mails.
inet_interfaces = all

######################################################################
#
#  FROM WHOM DO WE ACCEPT MAILS?
#
######################################################################

#  http://jimsun.linxnet.com/misc/restriction_order_prelim-03.txt says:
#
#	It was noted earlier that a match stops further processing of 
#       an access list, and of the restriction stage that "called" it.
#       But what does a restriction return when there's no match?  
#       Well, it returns "DUNNO."  ("I don't know, somebody else 
#       decide.")
#
#  and earlier says:
#
#	Postfix' restriction stages are as follows, and are processed
#       in the following order:
#	
#		smtpd_client_restrictions
#		smtpd_helo_restrictions
#		smtpd_sender_restrictions
#		smtpd_recipient_restrictions
#		smtpd_data_restrictions
#	
#	regardless of the order in which they're listed in main.cf.
#	
#	...
#	
#	Each restriction stage must evaluate to "OK" or "DUNNO" for 
#       processing to continue with the next stage.
#
#  So we're going to define a series of *_restrictions, which will be 
#  applied according to the above rules.
#
#  In fact, we need only one restriction: smtpd_recipient_restrictions.

#  Later we say that we accept mails from clients connecting to us from 
#  $mynetworks. Let that be derived from our interfaces' CIDRs.
mynetworks_style = subnet

#  Later we say that we accept mails from clients that do SASL 
#  authentication. Let us allow clients to authenticate this way 
#  (this still doesn't mean we'll accept their mails, only that we
#  allow them to authenticate) and say how we validate their attempts
#  to authenticate. smtpd_sasl_path is relative to /var/spool/postfix, but
#  cannot be absolute!
smtpd_sasl_auth_enable = yes 
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth

#  In addition, we can perform some simple checks on the client's 
#  self-introducing 'HELO'. From what I've read, 'HELO' seems pretty
#  pointless  but at the same time a lot of spammers either omit this
#  or set it to the that name of the server the client is connecting 
#  *to*! I'm going to try commenting this out to see what happens! It 
#  seems to be not very clear what happens when a client doesn't match
#  any of the restrictions. More info at
#  http://www.unixwiz.net/techtips/postfix-HELO.html.
#smtpd_helo_required = yes
#smtpd_helo_restrictions = permit_sasl_authenticated

#  And finally here is the list of who we will accept mails from
smtpd_recipient_restrictions = permit_sasl_authenticated, 
    permit_mynetworks, 
    reject_unauth_destination, 
    reject_rbl_client zen.spamhaus.org, 
    reject_rbl_client bl.spamcop.net, 
    check_policy_service inet:127.0.0.1:10023, 
    permit

######################################################################
#
#  IN-BOUND ENCRYPTION
#
######################################################################

smtpd_use_tls = yes
#  Offer encryption, but do not require it (requiring would contravene
#  RFC2487)
smtpd_tls_security_level = may
smtpd_tls_cert_file = /etc/ssl/certs/mail.pem
smtpd_tls_key_file = /etc/ssl/private/mail.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
#  Poodle protection
smtpd_tls_mandatory_protocols = !SSLv2 !SSLv3
smtpd_tls_protocols = !SSLv2 !SSLv3

######################################################################
#
#  OUT-BOUND ENCRYPTION
#
######################################################################

#  Gmail requires next two lines; others probably don't care
smtp_use_tls = yes
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
#  Poodle protection
smtp_tls_mandatory_protocols = !SSLv2 !SSLv3
smtp_tls_protocols = !SSLv2 !SSLv3
#  Our relay (gmail.com) requires us to to SASL authwenticate.
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options =

######################################################################
#
#  ADDRESS REWRITING
#
######################################################################

#  If an address matches *.$mydomain then change it to $mydomain. 
masquerade_domains = $mydomain
#  Which addresses to we masquerade?
masquerade_classes = envelope_sender, header_sender, 
    header_recipient, envelope_recipient
#  Adjust incomplete addresses for mails generated locally.
local_header_rewrite_clients = permit_mynetworks
#  Mail redirection for mail we've not yet decided is remote or local
#  (alias_maps is only for mail we've decided is for local delivery).
virtual_alias_maps = hash:/etc/postfix/virtual

######################################################################
#
#  WHICH DOMAINS ARE REMOTE AND WHICH ARE LOCAL?
#
######################################################################

#  Local is localhost, our private domain name and various public 
#  names we might be known as.
mydestination = localhost, $mydomain, MY_OTHERNAMES

######################################################################
#
#  HOW TO DO REMOTE DELIVERY
#
######################################################################

relayhost = RELAY_HOSTNAME
#  why?
smtp_host_lookup = native     

######################################################################
#
#  HOW TO DO LOCAL DELIVERY
#
######################################################################

#  Note aliases are consulted by LDA (so never consulted if everything
#  relayed)
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mailbox_size_limit = 0
mailbox_command = /usr/lib/dovecot/deliver

######################################################################
#
#  MISCELLANEOUS
#
######################################################################

recipient_delimiter = +
inet_protocols = ipv4
append_dot_mydomain = no
compatibility_level = 2
message_size_limit = 102400000
#debug_peer_level = 10
#debug_peer_list = 80.187.84.82
  4. Run: 
    rm -f /etc/mailname
echo "root: MY_ROOTRCPT" > /etc/aliases
> /etc/postfix/virtual
> /etc/postfix/sasl_passwd
[ $RELAY_AUTHENTICATE = yes ] && \
    echo "RELAY_HOSTNAME RELAY_LOGIN:RELAY_PASSWD" \
    >> /etc/postfix/sasl_passwd
  5. Replace markers: 
    perl -pi -e "s/MY_FQHN/$MY_FQHN/g; \
             s/RELAY_HOSTNAME/$RELAY_HOSTNAME/g; \
             s/RELAY_AUTHENTICATE/$RELAY_AUTHENTICATE/g; \
             s/RELAY_LOGIN/$RELAY_LOGIN/g;
           \ s/RELAY_PASSWD/$RELAY_PASSWD/g; \
             s/MY_OTHERNAMES/${MY_OTHERNAMES// /,}/g; \
             s/MY_ROOTRCPT/${MY_ROOTRCPT/@/\\@}/g" \
    /etc/postfix/main.cf /etc/aliases \
    /etc/postfix/virtual /etc/postfix/sasl_passwd
  6. Update some databases that use the files that had markers: 
    newaliases
postmap hash:/etc/postfix/virtual
postmap hash:/etc/postfix/sasl_passwd
service postfix stop
service postfix start
  7. When running a mail client at work that attempts to connect to port 25 on this mail server, it can be that work’s firewalls block access. Therefore we enable SMTPS:
    1. Edit /etc/postfix/master.cf and uncomment the line: 
      smtps     inet  n       -       y       -       -       smtpd
    2. Rerun: 
      service postfix restart
  8. Testing:
    1. Send yourself a mail with the SMTP server and the IMAP server being the new mailserver
    2. send a mail to another account that you can read.
  9. If appropriate, add an entry for webmaster to /etc/aliases and run: 
    newaliases
  10. If the system uses PCMS then /etc/postfix/main.cf should somehow be saved into the site-config module.

Switchover part #1

  1. Create a temporary key pair to allow ssh between the old mailserver and the new mailserver.
  2. On the old mailserver run: 
    systemctl stop postfix
systemctl stop dovecot
systemctl stop mailman
systemctl disable postfix
systemctl disable dovecot
systemctl disable mailman

    (Note that we don’t actually shut down the old mailserver because it is also the Jabber server, though that will be moved to another container in the future.)

  3. On the new mailserver run: 
    systemctl stop postfix
systemctl stop dovecot
  4. Rsync all mail from the old mailserver to the new mailserver with something like: 
    #  Do this on lagane!
cd $(mktemp -d)
for DIR in indexes maildir; do
    mkdir $DIR
    (
        cd $DIR
        rsync -a marille:/var/mail/$DIR/ ./
        shift-uid-gid . 200000 200000
        rsync -a ./ fiori:/var/lib/libvirt/mountpoints/mandala/mail/
        cd ..
    )
    rm -fr $DIR
done

    (There are a couple of reasons it is done in such a complicated manner: firstly, on my old mail server /var/mail/maildir was a symlink pointing to an automounted NFS filesystem; secondly the UID/GID shift needs to be applied.)

  5. If you need to change your DNS database (e.g. change target of ‘mail’ CNAME; change MX) then do so now.
  6. If you have rules on your firewall for incoming mail to update, then update them now.
  7. On the new mailserver run: 
    systemctl start dovecot
systemctl start postfix
  8. Testing:
    1. Send test mail from local user to local user
    2. Send test mail from local user to remote user
    3. Send test mail from remote user to local user
    4. Set up a cronjob and check that that can send mail correctly.
  9. A note to myself: Android K-9 needs to be told to connect to the SMTP server using STARTTLS on port 465.
  10. We will delay configuring the use of a Let’s Encrypt certificate until we have installed a web server.

    Mailman

    Mailman3, which is in Debian 11, is so different from Mailman 2.1, which was in Debian 10, that, for the moment, I am undecided whether to use it or not.

    Switchover part #2

    Because mailman is currently not in use, it means there there is no webserver running on the mailserver. This makes getting and renewing a certificate simpler but non-standard.

    1. Ensure that the front-end webserver to does not forward requests that it receives over https to the mail server (it is not needed).
    2. Ensure that the front-end webserver forwards requests that it receives over http to the mail server, with a configuration something like this: 
      <VirtualHost *:80>
    ServerName mail.pasta.freemyip.com
    ServerAdmin webmaster@dont-use-this-address
    CustomLog /var/log/apache2/mail.pasta.freemyip.com/mail.pasta.freemyip.com-access.log combined2
    ErrorLog /var/log/apache2/mail.pasta.freemyip.com/mail.pasta.freemyip.com-error.log
    LogLevel warn
    ServerSignature Off

    <Location />
        ProxyPass http://mail.pasta.net/
        ProxyPassReverse http://mail.pasta.net/
    </Location>
</VirtualHost>
    3. On the mail server install and run certbot as follows: 
      apt-get -y install certbot
certbot certonly --standalone -d  <fqdn-mail-clients-use-to-connect-to-mail-server>

      (The option --standalone will – whenever requesting/renewing a certificate – start a standalone internal webserver to allow letsencrypt.org authenticate the renewal request; /etc/cron.d/certbot will respect this option when it is used to initially request the certificate.)

    4. To aid testing, which we will do shortly, remove ~/.mutt_certificates.
    5. Locate the certificate and key generated by certbot: 
      find /etc/letsencrypt/live/ -name fullchain.pem -o -name privkey.pem
    6. Edit /etc/dovecot/dovecot.conf and change the following directives: 
      ssl_cert = <<path-of-fullchain.pem>
ssl_key = <<path-of-privkey.pem>

      (Note that that first ‘<‘ is not part of the metasyntactic variable; i.e. it really should be there.)

    7. Restart dovecot: 
      systemctl restart dovecot
    8. Test by running mutt; it should not ask if the certificate should be accepted.
    9. Edit /etc/postfix/main.cf and change the following directives: 
      smtpd_tls_cert_file = <path-of-fullchain.pem>
smtpd_tls_key_file = <path-of-privkey.pem>
    10. Restart postfix: 
      systemctl restart postfix

    See also