Let's Automate Let's Encrypt

Jul 3, 2016   #SSL  #Let's Encrypt  #HTTPS  #nginx  #docker  #Linux Journal 

This article was featured and first appeared in June 2016 Issue of Linux Journal

Introduction

HTTPS is a small island of security in this insecure world, and in this day and age, there is absolutely no reason to not have it on every website you host. Up until last year there was just a single last excuse: purchasing certificates was kind of pricey. That probably was not a big deal for enterprises; however, if you routinely host a dozen of websites, each of them with multiple subdomains, and have to pay for each certificate out of your own dear pocket - well, that could quickly become a burden.

No more excuses. Embrace Let’s Encrypt, free Certificate Authority that has officialy left Beta status this April.

Aside from being totally free, there is another special thing about Let’s Encrypt certificates: they don’t last long. Currently all certificates issued by Let’s Encrypt are valid for just 90 days, and we should expect that some day this term will be even shorter. Although this short lifespan definitely creates a much higher level of security, many people think of this as an inconvenience, and I’ve seen people going back from using Let’s Encrypt to buying certificates from commercial certificate authorities just for this reason.

Of course, if you are running multiple websites, having to manually renew several certificates every three months could quickly become a burden. Some day you’ll forget (and you will regret that forgetfulness). Let’s leave routine to computers, right?..

If you are using Apache under Debian-based distribution, then Let’s Encrypt already got you covered: there is libaugeas0 package available, and it is capable of both issuing and renewal of certificates. If, like me, you prefer Nginx and want to have zero downtime automatic certificate update with industrial-grade encryption - keep reading, I will show you how to get there.

Implementation

First things first, let’s set some assumptions and requirements:

  1. You are running nginx webserver / load balancer, and you are going to use it for TLS termination (that is a fancy, but technically correct way of saying “Nginx will handle all this HTTPS stuff).
  2. Nginx serves several websites, and you want HTTPS on all of them, and you are not going to pay a single dime.
  3. You also want to get the highest grade on industry standard for SSL tests - SSL Lab’s SSL server test.
  4. You do not enjoy the idea of running some not-so-well-sandboxed third-party code on your server, and you would rather have this code in a Docker container.
  5. Naturally, you are lazy (or experienced) enough, so you want to write some scripts that will re-issue all certificates way before they expire.
  6. I tested this code on Debian Jessie running nginx 1.6.2 and Docker 1.9.1; should also work on all other flavors. If you do not have docker-engine installed, follow instructions from this guide.

Now, check if your Nginx supports TLS:

sudo nginx -V

Usually it is supported by default and should yield:

TLS SNI support enabled

We also need a place to store certificates:

sudo mkdir -m 755 /etc/letsencrypt

Don’t sweat about permissions for this directory, certificates themselves will not be publicly accessible. Now you need to make a small change in your Nginx configuration. Create a new file /etc/nginx/letsencrypt.inc with the following contents:

	location ^~ /.well-known/acme-challenge/ {
	    root /tmp/letsencrypt/www;
	    break;
	}

Then find your “server” section in Nginx configuration and add the following line to each website you host:

	include /etc/nginx/letsencrypt.inc;

So the final result will look like:

	server {
	    listen 80;
	    server_name example.com www.example.com;
	    ...
		include /etc/nginx/letsencrypt.inc;
		...
	}

After saving both files ask nginx to reload configuration:

sudo /usr/sbin/nginx -t && sudo service nginx reload

Notice that we are only reloading Nginx configuration - and Nginx very well knows how to do it without dropping connections.

Now let’s go get some certificates

Needless to say that all domain names that you are going to issue certificates for should resolve to your server IP address - otherwise it would be possible to issue certificates for somebody else’s domain and use these certificates for man-in-the-middle attacks.

mkdir -p /tmp/letsencrypt/www

# make sure you have the latest version of this image,
# and not some pre-beta - those used to be notoriously buggy
docker pull quay.io/letsencrypt/letsencrypt:latest

docker run --rm -it --name letsencrypt \
    -v /etc/letsencrypt:/etc/letsencrypt \                                                                                                                   
    -v /tmp/letsencrypt/www:/var/www \
    quay.io/letsencrypt/letsencrypt:latest \
    auth --authenticator webroot \
    --webroot-path /var/www \
    --domain=example.com --domain=www.example.com \
    --email=admin@example.com

This will pull and start a new Docker image with official Let’s Encrypt client. As you can see, we share two data volumes between the host and the container:

  • /etc/letsencrypt for storing Let’s Encrypt configuration, all certificates and chains
  • /tmp/letsencrypt/www for communication between your server with Let’s Encrypt servers: webroot plugin that runs inside the container will create a temporary challenge file for each of your domains, then Let’s Encrypt validation servers will send HTTP request to ensure that you are really controlling this domain and this server. These files are temporary and needed only during issuing or renewing a certificate.

You will need to agree on TOS by pressing a button, and after several seconds your certificate is ready. If you have several subdomains, as in this example, you can enumerate all of them - that will result in one shared certificate issued for all of these subdomains. However, if you have several domains, it would be much more convenient to have a separate certificate for each of them - just repeat this last “docker run …” command for each domain you have (and thank me later if some day you will decide to move one of your domains to a different server).

As you can see, the procedure for obtaining certificate is painless and safe. Almost all heave work is done for you behind the scene, and if you had to deal with certificates using some other traditional certification authority, you will know exactly what I mean. Whatever runs inside the container can only access two directories on the server, and only while it runs. After you get all certificates it is safe to remove temporary directory:

rm -rf /tmp/letsencrypt

Back to Nginx configuration. Getting A+ grade from SSLLabs will require some additional effort. Create new Ephemeral Diffie-Hellman prime (if this is the first time you see this term - there is a good article):

sudo openssl dhparam -out /etc/pki/tls/private/dhparam.pem 4096

_Word of caution: if you absolutely need to support ancient versions of client software, for example Java 6 clients, you have to skip this step and you have to comment “ssldhparam” line on the following step. These old clients do not support Diffie-Hellman parameters longer than 1024 bytes and you have to make a choice between supporting these clients and security.

Have some hot beverage, it will take some minutes to generate. Add these lines to “http” section of /etc/nginx/nginx.conf:

http {
	...
	ssl_dhparam /etc/pki/tls/private/dhparam.pem;
	ssl_session_cache shared:SSL:10m;
	ssl_session_timeout 60m;
	...
}

Create a new file /etc/nginx/ssl_options.inc:

    ssl on;
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers "ECDH+AESGCM DH+AESGCM ECDH+AES256 DH+AES256 ECDH+AES128 DH+AES ECDH+3DES DH+3DES RSA+AESGCM RSA+AES RSA+3DES !aNULL !MD5 !DSS";
    # Enable HSTS (HTTP Strict Transport Security) for half a year
    add_header Strict-Transport-Security "max-age=15768000;includeSubDomains";

And create a new “server” section:

	server {
	    listen 443;
	    server_name example.com www.example.com;

	    include /etc/nginx/letsencrypt.inc;
    	include /etc/nginx/ssl_options.inc;

    	ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

		# enable OCSP stapling (https://en.wikipedia.org/wiki/OCSP_stapling) to speedup first connect
	    ssl_stapling on;
	    ssl_stapling_verify on;
	    ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;

        ...
    }

Word of warning: Strict-Transport-Security header will tell each visitor that you promise to always use HTTPS in the future. It is a one-way street, and once you set it, there is no way back - your visitor’s browser will remember your promise and insist on having HTTPS.

After making all of these changes reload Nginx configuration again:

	sudo /usr/sbin/nginx -t && sudo service nginx reload

At this point your website should have HTTPS up and running. Try to open https://www.example.com/ in browser and enjoy green lock sign in address line. To verify quality of encryption go to SSL Labs and submit your hostname for a check (usually it takes several minutes).

So, you have HTTPS, how about disabling HTTP? Go back to HTTP “server” section and make the following improvement:

	server {
	    listen 80;
	    server_name example.com www.example.com;
		include /etc/nginx/letsencrypt.inc;
		...
        if ($scheme = "http") {
                rewrite ^/(.*)$ https://$host/$1 permanent;
        }
		...
    }

This will redirect all traffic from HTTP to HTTPS, automatically bringing all clients to secure version of your website. Reload Nginx configuration to activate changes.

Automatic renewal

Now it’s time to automate certificate renewals. Current policy of “Let’s Encrypt” allows to request 5 certificate renewals for a domain within 7 days. That means it wouldn’t be wise (and wouldn’t make much sense either) to try renew certificates every day. On the other hand, leaving it for the last moment before expiration is also quite dangerous. Luckily, there is an easy way to renew these certificates only when they have less than 30 days before expiration. To me 30 days sound just right: that means my certificates will be reissued every 60 days on average, and if something fails afterwards - I will have a whole months to get involved and fix whatever is broken.

Create a script for renewal (I placed it in /root/update_keys.sh) with this contents:

#!/bin/bash

mkdir -p /tmp/letsencrypt/www

ADMIN_EMAIL=admin@example.com
HOSTNAME=$(hostname)

OUTPUT="$((docker run --rm -i --name letsencrypt \
    -v /etc/letsencrypt:/etc/letsencrypt \
    -v /tmp/letsencrypt/www:/var/www \
    quay.io/letsencrypt/letsencrypt:latest renew) 2>&1)"

if [[ $? -eq 0 ]]; then
    echo "${OUTPUT}" | grep -q "No renewals were attempted"
    if [[ $? -eq 0 ]]; then
        # all certificates have more than 30 days left - nothing to do
        exit 0
    fi
    echo "${OUTPUT}" | tr -Cd '[:print:]\n' \
        | mail -s "${HOSTNAME}: Let's Encrypt keys renewal - success" "${ADMIN_EMAIL}"
else
    echo "${OUTPUT}" | tr -Cd '[:print:]\n' \
        | mail -s "${HOSTNAME}: Let's Encrypt keys renewal - failed, exit code $?!" "${ADMIN_EMAIL}"
    exit 1
fi

# test config, reload if successfull
/usr/sbin/nginx -t &> /dev/null 
if [[ $? -ne 0 ]]; then
    echo 'please fix configfile problem' \
        | mail -s "${HOSTNAME}: nginx unable to reload" "${ADMIN_EMAIL}"
    logger "nginx has errors - not reloaded"
else
    service nginx reload
    logger "nginx reloaded"
fi

rm -rf /tmp/letsencrypt

Remember to assign proper access rights:

sudo chmod u+x /root/update_keys.sh

And create a crontab entry:

sudo crontab -e

with a line like this:

17 2 * * * /root/update_keys.sh

That will trigger execution of this update script at 2:17 every day. Update script will check if your certificates have more than 30 days left, and if they don’t it will attempt to renew all expirting certificates. Are you wondering why I used 2:17 am? Well, there is a simple explanation for that: almost everybody else did not. Most of the people when creating cronjobs use some simple values like 1:00 am, 2:00 am, 3:30 am, 4:15 pm… and that is a really, really bad choice if your cronjob is supposed to talk to an external service, because that means the service will experience maximum loads every once in a while. It is bad for service, and it is not good for you - the chance of getting a timeout is significantly higher if you send a request during these peak loads.

That means: for this job - please, please do not use round value, do not use my value; use some random value instead, and everything is going to be fine.

As you can see, Let’s Encrypt managed to make possible full automation of certificate maintanance - if you are using it right, it just works. They totally reinvented whole certification authority industry by converting what used to be boring and expensive into what turned out to be fun and free.