Configuring XMPP services (revision 3)

Introduction

This procedure describes how Alexis Huxley set up a XMPP server at home. That this is at home is important: I have several servers using private IP addresses, but my ISP allocates one internet-accessible IP address to my router. This procedure is designed with that in mind.

If you do not want support for BOSH then skip all sections with BOSH in their titles. Here is more information about BOSH, but think of it as XMPP over HTTP(S).

If you do need support for an Apache reverse proxy then skip all sections with ‘without an Apache reverse proxy’ in their titles.

Conversely, if you don’t need support for an Apache reverse proxy then skip all sections with ‘with an Apache reverse proxy’ in their titles.

All commands are to be run as root unless specified otherwise.

Environment variables

The procedure refers to the following environment variables and assumes they are already set, so be sure to set them in a timely fashion.

  1. Set some environment variables: 
    ALLOW_REGISTRATION=false                         #  my server is not for public use
FRONTENDVHOST=<xmpp-server-public-name>          #  e.g. FRONTENDVHOST=jabber.pasta.freemyip.com
FRONTENDVPROT=http
FRONTENDVPORT=80
  2. If there is a reverse proxy then set some more environment variables: 
    HTTP_REDIRECTS_TO_HTTPS=false                    #  cert needed on backend, so certbot runs on backend, so http on backend must be reachable
FRONTENDVHOST_RUNS_CERTBOT=false                 #  needs special opts so not done by sub-procedure
FRONTENDHOST=<apache-reverse-proxy-local-name>   #  e.g. FRONTENDHOST=armonie.pasta.net
BACKENDHOST=<xmpp-server-local-name>             #  e.g. BACKENDHOST=jaborini.pasta.net
BACKENDVHOST=$BACKENDHOST
BACKENDVPROT=http
BACKENDVPORT=80

Initial installation of Prosody

  1. Run: 
    apt -y install prosody prosody-modules
systemctl stop prosody
  2. Stop Prosody from attempting to use IPv6 by editing /etc/prosody/prosody.cfg.lua and setting: 
    interfaces = { "*" }
local_interfaces = { "*" }
  3. I had very bad performance transferring files, so I stopped Prosody from applying limits by editing /etc/prosody/prosody.cfg.lua and commenting out: 
    ...
modules_enabled = {
    ...
    -- "limits";
    ...
  4. Remove the localhost configuration as this is not needed with: 
    sed -i -r 's/^(VirtualHost "localhost")/-- \1/' /etc/prosody/prosody.cfg.lua
rm -f /etc/prosody/conf.d/localhost.cfg.lua

Overview of Prosody/XMPP without Apache reverse proxy

The sections below with “Prosody/XMPP” in their titles are intended to establish this configuration without a reverse proxy:

Note that:

  • XMPP clients and servers connect to the prosody process on different ports
  • XMPP clients and servers require that the prosody process encrypt communications using an SSL certificate
  • we get the SSL certificate from letsencrypt.org
  • that necessitates having certbot listening on port 80 on the prosody box to respond to ownership challenges from letsencrypt.org
  • the router is required to forward all incoming connections to the right box
  • from the internet, both XMPP clients and web clients (specifically Lets Encrypt SSL certificate challenger) will use the name $FRONTENDVHOST: it is up to the router, based on the port, to direct XMPP traffic to the Prosody box and to direct HTTP traffic to certbot
  • from the home network, only XMPP clients will use the name $FRONTENDVHOST; it is up to the local DNS to direct (XMPP) traffic to the Prosody box.

Firewall for Prosody/XMPP without Apache reverse proxy

  1. Set up the following rules on the firewall:
    firewall portdestination hostdestination portpurpose
    5222prosody box5222client to server XMPP traffic
    5269prosody box5269server to server XMPP traffic
    80prosody box80Lets Encrypt SSL certificate challenge

Overview of Prosody/XMPP with Apache reverse proxy

Note that:

  • See the notes in the ‘Overview of Prosody/XMPP without Apache reverse proxy’ section above
  • it is up to the router, based on the port, to direct XMPP traffic to the Prosody box and to direct HTTP traffic to Apache reverse proxy

Firewall for Prosody/XMPP with Apache reverse proxy

  1. Set up the following rules on the firewall:
    firewall portdestination hostdestination portpurpose
    5222prosody box5222client to server XMPP traffic
    5269prosody box5269server to server XMPP traffic
    80Apache reverse proxy80Lets Encrypt SSL certificate challenge

    DNS for Prosody/XMPP without Apache reverse proxy

    1. Ensure that from the internet, $FRONTENDVHOST resolves to the IP address of router’s interface on the internet (the firewall will then forward both XMPP and HTTP traffic to the prosody box).
    2. Ensure that from the home network, the same name resolves to the IP address of the prosody box (some hints on how to do this can be found here).

    DNS for Prosody/XMPP with Apache reverse proxy

    1. Ensure that from the internet, $FRONTENDVHOST resolves to the IP address of router’s interface on the internet (and the firewall will then forward XMPP traffic to the prosody box and HTTP traffic to the Apache reverse proxy).
    2. Ensure that from the home network, the same name resolves to the IP address of the prosody box (some hints on how to do this can be found here).

    Certificates for Prosody/XMPP without Apache reverse proxy

    1. Run: 
      apt -y install certbot
    2. Run certbot. 
      certbot certonly --standalone -d $FRONTENDVHOST

      (The option --standalone option will – whenever requesting/renewing a certificate – start a standalone internal webserver to allow letsencrypt.org authenticate the renewal request; /etc/cron.d/certbot will respect this option when it is used to initially request the certificate.)

    Certificates for Prosody/XMPP with Apache reverse proxy

    1. Log in to the Apache reverse proxy.
    2. Complete configuring a reverse proxy (revision 1).
    3. Log in to the prosody box.
    4. Run: 
      apt -y install certbot
    5. Run certbot. 
      certbot certonly --standalone -d $FRONTENDVHOST

      (The option --standalone option will – whenever requesting/renewing a certificate – start a standalone internal webserver to allow letsencrypt.org authenticate the renewal request; /etc/cron.d/certbot will respect this option when it is used to initially request the certificate.)

      Configuring Prosody/XMPP

      1. Create a minimal new virtual host by running: 
        cat > /etc/prosody/conf.avail/$FRONTENDVHOST.cfg.lua <<EOF
VirtualHost "$FRONTENDVHOST"
ssl = {
    certificate = "/var/lib/prosody/$FRONTENDVHOST.fullchain.pem";
    key         = "/var/lib/prosody/$FRONTENDVHOST.privkey.pem";

}
allow_registration = "$ALLOW_REGISTRATION";
EOF
ln -sr /etc/prosody/conf.avail/$FRONTENDVHOST.cfg.lua /etc/prosody/conf.d
      2. Certbot has put the certificate and key in /etc/letsencrypt/live/$FRONTENDVHOST/. Certbot supports hooks to inform other programs about the new certificate at certificate renewal time. prosodyctl --root cert import <cert-dir>, run as root (as certbot is and so its hooks are as well) should have the rights to read the well-protected certificate directory: 
        jaborini# prosodyctl --root cert import /etc/letsencrypt/live/jabber.pasta.freemyip.com
Imported certificate and key for hosts jabber.pasta.freemyip.com
jaborini#

        However, it does not copy the certificates to where the configuration file says they are to be loaded from, so it is clearer to simply copy the files and reload prosody.  So now we create a simple script to run this at renewal time:

        cat > /etc/letsencrypt/renewal-hooks/deploy/prosody <<EOF
#!/bin/bash
set -e

#  Copy certificate to where prosody can read it.
cp /etc/letsencrypt/live/$FRONTENDVHOST/fullchain.pem /var/lib/prosody/$FRONTENDVHOST.fullchain.pem
cp /etc/letsencrypt/live/$FRONTENDVHOST/privkey.pem   /var/lib/prosody/$FRONTENDVHOST.privkey.pem
chown prosody:prosody /var/lib/prosody/$FRONTENDVHOST.fullchain.pem
chown prosody:prosody /var/lib/prosody/$FRONTENDVHOST.privkey.pem
systemctl reload prosody
EOF
chmod 755 /etc/letsencrypt/renewal-hooks/deploy/prosody

        (Note that although no commands in this script require bash as the execution shell yet, later in this procedure commands will be added that do.)

      3. If migrating to a new server then copy over account information, which is stored in /var/lib/prosody/{accounts,roster,vcard}, remembering to set the owner & group correctly.
      4. Restart the service by running: 
        service prosody restart
      5. Verify that the certbot plugin works by running: 
        /etc/letsencrypt/renewal-hooks/deploy/prosody

      Creating accounts

      1. Run: 
        echo $ALLOW_REGISTRATION

        and if that displays true then skip this section.

      2. IAs root run: 
        prosodyctl adduser <username>@$FRONTENDVHOST

      Configuring Pidgin with XMPP

       

      1. Select Accounts –> Manage Accounts –> Add.
      2. Fill in the details using $FRONTENDVHOST, as shown in this example:
      3. Note that:
        • Domain is the value of $FRONTENDVHOST
        • Resource is a description of the client’s location
        • depending on whether registration is allowed or not, you either provide a password or click ‘Create this new account on the server’
        • Set ‘Remember password’ and ‘Use this buddy icon …’ as you see fit.
      4. Add yourself as a buddy (this is helpful for testing) and then send yourself a message! 🙂

      Overview of Prosody/BOSH without Apache reverse proxy

        Firewall for Prosody/BOSH without Apache reverse proxy

        1. The firewall needs to allow:
          firewall portdestination hostdestination portpurpose
          5280prosody box5280client to server BOSH-over-http traffic
          5281prosody box5281client to server BOSH-over-https traffic
          80prosody box80SSL certificate renewal challenge

        Certificates for Prosody/BOSH without Apache reverse proxy

        The SSL certificate is already on the prosody box as a result of completing “Certificates for Prosody/XMPP” above.

        Configuring Prosody/BOSH without Apache reverse proxy

        Note that some of this is indentical to the section ‘Configuring Prosody/BOSH with Apache reverse proxy’ below.

        1. Edit /etc/prosody/prosody.cfg.lua uncomment the bosh module and tell bosh where the certificate is: 
          ...
modules_enabled = {
    ...
    "bosh"; 
    ...
}
...
https_ssl = {
    certificate = "/var/lib/prosody/jabber.pasta.freemyip.com.fullchain.pem";
    key = "/var/lib/prosody/jabber.pasta.freemyip.com.privkey.pem";
}

          but with the correct hostname. Note that it is correct that this is specified in the global section.

        2. Run: 
          systemctl restart prosody
        3. Test as follows:
          1. Use ss to verify the listening ports, as in this example: 
            jaborini# ss -tlpn | grep 52
LISTEN 0      128          0.0.0.0:5222      0.0.0.0:*    users:(("lua5.4",pid=3462,fd=13))
LISTEN 0      128          0.0.0.0:5281      0.0.0.0:*    users:(("lua5.4",pid=3462,fd=12))
LISTEN 0      128          0.0.0.0:5280      0.0.0.0:*    users:(("lua5.4",pid=3462,fd=11))
LISTEN 0      128          0.0.0.0:5269      0.0.0.0:*    users:(("lua5.4",pid=3462,fd=8)) 
jaborini#
          2. Use w3m to access the BOSH url over http and https, as in this example: 
            damson$ w3m -dump http://$FRONTENDVHOST:5280/http-bind
Prosody BOSH endpoint

It works! Now point your BOSH client to this URL to connect to Prosody.

⚠ This endpoint is not considered secure! ⚠

damson$ w3m -dump https://$FRONTENDVHOST:5281/http-bind
Prosody BOSH endpoint

It works! Now point your BOSH client to this URL to connect to Prosody.

damson$

            Note that that the output from the first command  (“It works! … not secure!” and the warning triangles) is very misleading! It does not work because the server won’t to talk on an unencrypted connection. If you really want to support BOSH/http (i.e. unencrypted) then modify /etc/prosody/prosody.cfg.lua and add the following line:

            consider_bosh_secure = "true";

            and restart prosody. (There is a valid use case for this option: when the client’s connection to the reverse proxy is encrypted and the reverse proxy’s connection to prosody is not.)

        Configuring Pidgin with BOSH/http over the standard BOSH/http port

        1. Depending on whether you have already created your account or not then either add or modify your account (see above for details) and then go to the Advanced tab.
        2. Fill in the details using $FRONTENDVHOST, as shown in this example:
        3. Note that:
          • BOSH URL: is http://$FRONTENDVHOST:5280/http-bind
          • Connection port: 0 (the real port number must be encoded in the BOSH URL!!!)
          • Connection security: If set to ‘Require encryption’ (the default) then pidgin will report “You require encryption, but no TLS/SSL support was found” and will not connect.
        4. Connect and send yourself a test message.

        Configuring Pidgin with BOSH/https over the standard BOSH/https port

        1. Depending on whether you have already created your account or not then either add or modify your account (see above for details) and then go to the Advanced tab.
        2. Fill in the details using $FRONTENDVHOST, as shown in this example:
        3. Note that:
          • BOSH URL: is https://$FRONTENDVHOST:5281/http-bind
          • Connection port: 0 (the real port number must be encoded in the BOSH URL!!!)
        4. Connect and send yourself a test message.

        Overview of Prosody/BOSH with Apache reverse proxy

        In principle, this is what we want:

          Note that:

          • Apache must decide whether to forward an incoming request to prosody or to certbot (the ProxyPass directive can do this)
          • Certbot could be run on the reverse proxy, but I choose to run it on the prosody box, in simpler configurations, only the prosody box needs a certificate

          Firewall for Prosody/BOSH with Apache reverse proxy

          1. Since there are other vhosts, it is assumed that the firewall already forwards traffic on 80 and 443 to the Apache reverse proxy, so no new firewall rules are required.
          2. However, depending on whether BOSH fulfils all your needs or not, then following rules could be removed:
            firewall portdestination hostdestination portpurpose
            5222prosody box5222client to server XMPP traffic

          Certificates for Prosody/BOSH with Apache reverse proxy

          The SSL certicate needs to be installed on two hosts:

          1. the prosody box (because the prosody process will want to provide SSL connections for clients connecting to $FRONTENDVHOST:5222, $FRONTENDVHOST:5269)
          2. the reverse proxy (because it is the end point of a client’s connection to $FRONTENDVHOST:443)

          As explained above, I decided that the prosody box should manage certificate renewal because in simpler configurations, only the prosody box needs a certificate (though this isn’t much of a reason). This means:

          1. the prosody box must run certbot (we already did this as a certificate is required for prosody/XMPP)
          2. the reverse proxy, listening on port 80 must:
            • send BOSH/http traffic to the prosody server listening on port 5280 on the prosody box
            • send SSL challenges to certbot listening on port 80 on the prosody box (remember: earlier we ran certbot --standaone) on port 80 on the prosody box

            (we could do this with ProxyPassMatch rules), or:

            • send all traffic to certbot listening on port 80 on the prosody (and BOSH/http won’t be supported)
          3. certbot on the prosody box must:
            • copy the updated certificate to where the prosody server can read it and tell the prosody server to reload it (we already set up a certbot hook to do this several sections earlier)
            • copy the updated certificate to to the reverse proxy (we could extend the certbot hook to do this)

            SSL challenges look like this:

            armonie# sed -nr 's/.*"(GET \/\.well-known\/[^ ]*).* 200 .*/\1/p' /var/log/apache2/*/*.log | uniq | head -3
GET /.well-known/acme-challenge/6hDPcYg-SZK9_Y7gXUhqmN2S74yrgjyEpHFR1JeQkk8
GET /.well-known/acme-challenge/IE0rE9Nq_m2Wv3AyJjWihuTyqTyqQOW-0iF9zPPYHto
GET /.well-known/acme-challenge/eqWUF9ICLXteTSwCn3dfTd_nsODt7YGOmDsUuk6vR3Y
armonie#

            So, BOSH traffic and SSL challenges can easily be distinguished.

            1. Currently the reverse proxy forwards all requests on port 80 to certbot on the prosody box because of these directives in /etc/apache2/sites-available/$FRONTENDVHOST.conf: 
              ProxyPass        /                            http://jaborini.pasta.net:80/
ProxyPassReverse /                            http://jaborini.pasta.net:80/

              (jaborini.pasta.net is my Prosody box.)

              However, we need to restrict the scope of this proxing so that only SSL challenges go to certbot. So change the above lines to something like this:

              ProxyPass        /.well-known/acme-challenge/ http://jaborini.pasta.net:80/.well-known/acme-challenge/
ProxyPassReverse /.well-known/acme-challenge/ http://jaborini.pasta.net:80/.well-known/acme-challenge/
            2. Additionally, to avoid non-matching URLs listing /var/www/html, run: 
              chmod 000 /var/www/html
            3. Run: 
              systemctl reload apache2
            4. To test do the following:
              1. On the prosody box run a simple web server using socat: 
                socat -v -d -d TCP-LISTEN:80,crlf,reuseaddr,fork SYSTEM:"
    echo HTTP/1.1 200 OK; 
    echo Content-Type\: text/plain; 
    echo; 
    echo \"this is fake certbot replying\";
"

                (This command was based on the one here.)

              2. From somewhere on the internet run these two w3m commands and note the output: requests that don’t look like SSL challenges are rejected because the bosh module is not listening on 5280; requests that do look like SSL challenges are accepted and socat replies: 
                damson$ w3m -dump http://$FRONTENDVHOST/
Forbidden

You don't have permission to access this resource.

damson$ w3m -dump http://$FRONTENDVHOST/.well-known/acme-challenge/blah
this is fake certbot replying
damson$

                Note that that first request never makes it as far as the prosody box: it reaches the correct vhost, but as that has no DocumentRoot defined, it tries to look in /var/www/html, which it does not have permission to do.

              3. Kill the socat process.
            5. Set up the ssh access from the prosody box to the reverse proxy as follows:
              1. On the prosody box run: 
                ssh-keygen -qt ed25519 -N "" -f ~/.ssh/id_ed25519_certxfer
              2. Append the just-generated public key to root’s ~/.ssh/authorized_keys on the reverse proxy. (Note to myself: remember that PCMS will reset this file, so the key needs to be added to gen-facility-local).
              3. Test by calling ssh as in the following example: 
                jaborini# ssh -i ~/.ssh/id_ed25519_certxfer $FRONTENDHOST hostname
armonie
jaborini#
            6. Run the following commands to add the lines necessary to make the  existing certbot hook copy the renewed SSL certificate over the reverse proxy: 
              cat >> /etc/letsencrypt/renewal-hooks/deploy/prosody <<EOF

#  Copy XMPP/SSL certificate to reverse-proxy so it can use it for BOSH/https.
rsync -e "ssh -i ~/.ssh/id_ed25519_certxfer" -va --delete /etc/letsencrypt/live/$FRONTENDVHOST/    $FRONTENDHOST:/etc/letsencrypt/live/$FRONTENDVHOST/
rsync -e "ssh -i ~/.ssh/id_ed25519_certxfer" -va --delete /etc/letsencrypt/archive/$FRONTENDVHOST/ $FRONTENDHOST:/etc/letsencrypt/archive/$FRONTENDVHOST/
{
    date
    echo
    echo "This certificate was updated by a certbot hook running on \$(hostname -f)!"
    echo "See https://www.pasta.freemyip.com/computing/procedures/configuring-xmpp-services/configuring-xmpp-services-revision-3/."
} | ssh -i ~/.ssh/id_ed25519_certxfer $FRONTENDHOST "cat > /etc/letsencrypt/live/$FRONTENDVHOST/README.Alexis"
ssh -n -i ~/.ssh/id_ed25519_certxfer $FRONTENDHOST systemctl reload apache2
EOF

              Note that:

              • since we are appending these lines to an existing hook that first line is with >>
              • it’s questionable if we should copy the certificate to /etc/letsencrypt/ on the reverse proxy, as, althrough the key was generated by certbot, it is not managed by certbot on the reverse proxy
            7. Edit /etc/letsencrypt/renewal-hooks/deploy/prosody and make really really sure that the target vhost name is set correctly (i.e. that you didn’t mistype the ‘FRONTENDHOST’ variable name); otherwise all your certificates will be removed!
            8. Test the hook by calling it directly: 
              /etc/letsencrypt/renewal-hooks/deploy/prosody || echo FAILED

              and, if everything is ok, then you can remove the -v option on the two rsync calls.

            Configuring Prosody/BOSH with Apache reverse proxy

            1. Complete ‘Configuring Prosody/BOSH without Apache reverse proxy’ above.
            2. Having already completed ‘Certificates for Prosody/BOSH with Apache reverse proxy’ above, the the proxying of port 80 on the reverse proxy through to port 5280 on the prosody box is almost configured. We only need to add the following lines to /etc/apache2/sites-available/$FRONTENDVHOST.conf after the already-present ProxyPass lines: 
              ProxyPass        / http://jaborini.pasta.net:5280/ 
ProxyPassReverse / http://jaborini.pasta.net:5280/

              It is essential that these lines are after the already-present ProxyPass lines: first match wins so longer URLs should go first!

            3. Run: 
              FRONTENDVPROT=https
FRONTENDVPORT=443
BACKENDVPROT=https
BACKENDVPORT=5281

              (All other variable settings are still applicable.)

            4. Complete configuring a reverse proxy (revision 1).
            5. Edit /etc/apache2/sites-available/$FRONTENDVHOST-ssl.conf and comment out the snake-oil-certificate-related lines and uncomment the letsencrypt-certificate-related lines. (We can do this without just having applied for a letsencrypt certificate because the prosody box’s certbot’s hook has taken care of copying it over.)
            6. Since the reverse proxy not only connects to $BACKENDVHOST, but also uses that name in its requests and prosody doesn’t know that it may be connected to using that name, there is a mismatch between the names being used, which will result in prosody reporting Unknown host: jaborini.pasta.net . There are two ways to solve this: one is to use the http_host assignment in /etc/prosody/conf.avail/$FRONTENDVHOST.cfg.lua (see here) but a nicer one is just to tell Apache to tell connect to one host, but use a different name in its requests. Add this to the http vhost configuration file: 
              RequestHeader set Host "jabber.pasta.freemyip.com"
RequestHeader set X-Forwarded-Proto: http
ProxyPreserveHost On

              and this to the https configuration file:

              RequestHeader set Host "jabber.pasta.freemyip.com"
RequestHeader set X-Forwarded-Proto: https
ProxyPreserveHost On
            7. Run: 
              systemctl reload apache2

            Configuring Pidgin with BOSH/http over the standard (non-BOSH) http port

            Remember:

              1. this will only work if you set: 
                consider_bosh_secure = "true";
              2. if using an Apache reverse proxy, then DNS will need changing in order direct clients to that rather than directly to the prosody box.
            1. Depending on whether you have already created your account or not then either add or modify your account (see above for details) and then go to the Advanced tab.
            2. Fill in the details using $FRONTENDVHOST, as shown in this example:
            3. Note that:
              • BOSH URL: is http://$FRONTENDVHOST:80/http-bind
              • Connection port: 0 (the real port number must be encoded in the BOSH URL!!!)
              • Connection security: If set to ‘Require encryption’ (the default) then pidgin will report “You require encryption, but no TLS/SSL support was found” and will not connect.
            4. Connect and send yourself a test message.

            Configuring Pidgin with BOSH/https over the standard (non-BOSH) https port

            Remember:

              1. if using an Apache reverse proxy, then DNS will need changing in order direct clients to that rather than directly to the prosody box.
            1. Depending on whether you have already created your account or not then either add or modify your account (see above for details) and then go to the Advanced tab.
            2. Fill in the details using $FRONTENDVHOST, as shown in this example:
            3. Note that:
              • BOSH URL: is https://$FRONTENDVHOST:443/http-bind
              • Connection port: 0 (the real port number must be encoded in the BOSH URL!!!)
            4. Connect and send yourself a test message.

            See also

            1. Computing
            2. Download Prosody
            3. XMPP Compliance Tester
            4. converse.js