Using Fabric To Fix Heartbleed On Ubuntu 12.04

This morning I wrote two small Fabric tasks to help deal with Heartbleed (aka CVE-2014-0160) on Ubuntu 12.04, and I thought I would share in case they can help anybody out. If you've never used Fabric before, you can clone my repo of random tasks which should help you get started with some examples.

hb_checkversions

The first task is a scanner that will check the libssl package version and report if it's vulnerable. For Precise, the package with the fix is libssl 1.0.1-4ubuntu5.12. You can check your installed version manually by running apt-cache policy libssl1.0.0.

# fabfile.py
from fabric.api import *

@task
@parallel(pool_size=3)
def hb_checkversions():
    with settings(
            hide('everything'),
            quiet=True,
    ):
        env.eagerly_disconnect = True
        pkg_info = run(
            "dpkg-query -W -f='${status}|${version}' libssl1.0.0", quiet=True
        )
        if pkg_info.failed or 'install ok installed' not in pkg_info.stdout:
            print "[{}] NOT FOUND: {}".format(env.host_string, pkg_info.stdout)
            return
        else:
            version = pkg_info.stdout.split('|')
            build_date = run('openssl version -b')
            if version[1].rpartition('.')[2] < '12':
                print "[{}] VULNERABLE: libssl {} {}".format(
                    env.host_string, version[1], build_date.stdout
                )
            else:
                print "[{}] OK: libssl {} {}".format(
                    env.host_string, version[1], build_date.stdout
                )

Running it looks like this:

(fabrictasks)pliu@behemoth:~/projects/fabrictasks$ fab hb_checkversions --hide=running

[host1] OK: libssl 1.0.1-4ubuntu5.12 built on: Mon Apr  7 20:33:29 UTC 2014
[host2] OK: libssl 1.0.1-4ubuntu5.12 built on: Mon Apr  7 20:33:29 UTC 2014
[host3] VULNERABLE: libssl 1.0.1-4ubuntu5.11 built on: Wed Jan  8 20:45:51 UTC 2014
...

hb_update

The second task updates OpenSSL and restarts Nginx or Apache if installed. It's purposely verbose because I wanted to manually monitor the process, but you can tone it down by capturing and analyzing stdout instead of just printing it out.

# fabfile.py
from fabric.api import *

@task
def hb_update():
    env.eagerly_disconnect = True
    pkg_info = run(
        "dpkg-query -W -f='${status}|${version}' libssl1.0.0", quiet=True
    )
    if pkg_info.failed or 'install ok installed' not in pkg_info.stdout:
        print "libssl not found"
        return
    else:
        # bail if version is okay
        version = pkg_info.stdout.split('|')
        build_date = run('openssl version -b')
        if version[1].rpartition('.')[2] >= '12':
            print "libssl version ok: {} {}".format(
                version[1], build_date.stdout
            )
            return

        # update
        print "Updating..."
        sudo(
            'apt-get install -qq --only-upgrade openssl libssl1.0.0 > /dev/null'
        )

        # look for nginx or apache, restart if found
        pkg_info = run(
            "dpkg-query -W -f='${package} ${status}|' nginx apache2",
            quiet=True
        )
        for res in pkg_info.stdout.split('|'):
            if 'nginx install ok installed' in res:
                print "Nginx found, attempting restart"
                sudo('service nginx restart')
            if 'apache2 install ok installed' in res:
                print "Apache found, attempting restart"
                sudo('service apache2 restart')

Running it looks like this:

(fabrictasks)pliu@behemoth:~/projects/fabrictasks$ fab hb_update -I --hosts=host1

Upgrading OpenSSL
[host1] sudo: apt-get install -qq --only-upgrade openssl libssl1.0.0 > /dev/null
[host1] out: sudo password:
[host1] out: 
Nginx found, attempting restart
[host1] sudo: service nginx restart
[host1] out: sudo password:
[host1] out: Restarting nginx: nginx.
[host1] out: 
Disconnecting from host1... done.

Verification and Cleanup

You can verify OpenSSL is fixed by running openssl version -a. The build date should be Mon Apr 7 20:33:29 UTC 2014 or later.

root@host1:~# openssl version -a
OpenSSL 1.0.1 14 Mar 2012
built on: Mon Apr  7 20:33:29 UTC 2014
...

After updating OpenSSL, remember to restart any processes that are using it (like stunnel, syslog-ng, puppet, etc) and to regenerate SSL keys which may have been compromised. To get a list of processes which need to be restarted, run lsof | grep ssl | grep DEL as root. Just reboot the whole system if you aren't sure about this part.

root@host1:~# lsof | grep ssl | grep DEL
whoopsie    841   whoopsie  DEL       REG              252,0                  924 /lib/x86_64-linux-gnu/libssl.so.1.0.0
syslog-ng 13349       root  DEL       REG              252,0                  924 /lib/x86_64-linux-gnu/libssl.so.1.0.0
puppet    24954       root  DEL       REG              252,0                  924 /lib/x86_64-linux-gnu/libssl.so.1.0.0

In addition to regenerating SSL keys, consider what other sensistive data may have been leaked. Passwords, tokens, cookies, anything you send over the wire. It should all be considered compromised and reset accordingly.

Man, what a depressing topic to start a blog with.