Developing With Docker At 500px, Part One

Note: I originally wrote this for the 500px engineering blog.

This article is part one of two.


Computers are hard. Distributed systems are harder. That's a popular thing people say, right? Well, it's true.

The 500px system architecture can be thought of as a large Rails monolith surrounded by a constellation of Go and Ruby microservices. The monolith houses our main web app and the 500px API, and the microservices provide ancillary functionality like image processing, search services, user feeds and activity streams, push notifications and other background processing. While we love the microservices pattern because it offers us lots of technical freedom and flexibility, it can make development pretty painful sometimes.

We're in the process of trying to make local development a bit easier with Docker, so we thought we'd blog a bit about our journey so far. This article is the first in a series about how we are planning to address the problem, the challenges involved, and the (hopefully successful) results of our experiment.

Please note, this article assumes a passing familiarity with Docker, Docker Compose and continous delivery concepts.

The Problem

At the time of this writing, all development of the 500px monolith is done locally. To get the monolith fully up and running, developers need to run an instance of every microservice we have and all of their related backend datastores. This involves installing lots of libraries, running and compiling tons of Go programs, and setting up a bunch of configuration heavy things like MySQL, MongoDB, RabbitMQ, Elasticsearch and Roshi (a Redis backed CRDT database for time series event storage). All of this complexity is the cause of a lot of cognitive overhead and headaches for the team, and it makes onboarding new developers sometimes more difficult than it should be.

Things move pretty fast around here and we deploy frequently, which means debugging a local dev stack can get pretty hairy. Depending on the context, a simple question like "why doesn't this upload work" may have a simple answer, but it also runs the chance of having an incredibly complicated answer that requires deep knowledge about the stack, 45 minutes, a whiteboard, Charles, pprof, tcpdump, and maybe a couple beers.

The situation is generally manageable at the moment, but as we scale our team in every direction and across multiple offices, things are starting to get out of hand. It's time to step up our local dev game.

Docker To The Rescue

The best solution to this problem is, we think, Docker. Instead of asking developers to compile and manage their own versions of microservices, our plan is to containerize everything we can and then ship the whole stack as images to our dev teams. To automate setting up, updating and tearing down these images, we're using Docker Compose (and, probably, a Bash script or two). If we can successfully containerize the backend stack and make it easy to deploy, working on the monolith should become much easier.

The ideal implementation is such that a brand new developer can show up at 500px, unwrap their shiny new computer and, with no context and minimal instruction, simply install Docker and get the entire stack up and running within an hour or so. Ideally, they will also push a change and deploy the site that same day, but we've already got that part covered. And when some part of the system changes the next day, that same developer should be able to simply pull down an updated container and keep on truckin'.

It's a simple idea, but a powerful one.

Dockerfiles And Continuous Integration

The first step of this project is one we have already taken and are pretty happy with: building containers and integrating Docker into our CI pipeline.

Building Containers

Our Go apps are all pretty straightforward, so containerizing them was relatively easy. Since we already package everything as Debian packages, most of our dockerfile's just look something like this:

FROM ubuntu-debootstrap:precise

# Install repo
RUN apt-get -qq update && \
    apt-get -qq install curl apt-transport-https --no-install-recommends && \
    curl ${REPO_KEY} | apt-key add - && \
    echo "deb ${REPO_URL} precise main" >> /etc/apt/sources.list && \
    rm -rf /var/lib/apt/lists/* && rm -rf /usr/share/man/* && rm -rf /usr/share/doc/*

# Install service
RUN apt-get -qq update && \
    apt-get -qq install ${PACKAGE_NAME} --no-install-recommends && \
    rm -rf /var/lib/apt/lists/* && rm -rf /usr/share/man/* && rm -rf /usr/share/doc/*

# Entrypoint
WORKDIR service-dir

The resultant container is totally stateless, and service configuration is done though command line arguments or shell environment variables. Since we are going to use Docker Compose, we can set up that stuff in docker-compose.yml or the user's shell env.

A Side Note About Secrets In Dockerfiles

We host all our Debian packages in private repositories which require API keys to access. Unfortunately, since Docker doesn't support build-time argument passing, we've had to commit repo access keys to our dockerfiles and thus, to version control. This is a huge bummer and it makes us very sad, but there wasn't any way around it.

Secrets in Docker are, in general, pretty tough to deal with at the moment, but we have hope that this will get easier in future.

Continuous Integration

After writing some dockerfiles, next up was integrating Docker image building into our continuous integration pipeline.

Our pipeline consists of four services: Github for hosting code, Travis CI for CI, for hosting packages, and Docker Hub for hosting Docker images. We like all of these tools and think they are rad.

From keyboard to container, the pipeline looks like this:

  1. A service maintainer writes some code, pushes to Github, and merges their branch to master
  2. Travis runs tests, builds master, and creates a .deb
  3. Travis uploads the .deb to Packagecloud
  4. Travis then does a POST to activate a Docker Hub build webhook
  5. Docker Hub builds the image, using the .deb that we just uploaded
  6. A service consumer runs docker pull and downloads the new image

Easy as 123... 456.

Curious readers might be wondering why we didn't simply use Docker Hub's Automated Build feature. The reason we set up our pipeline this way was to avoid building our projects from source, inside the Docker image. Building the project in the image is simpler, but would require installing Go and other build time dependencies, which results in a slow build and a fat gross image.

Our Debian packages on the other hand contain only a compiled Go binary and whatever libs we need to deploy. By not installing build time cruft, we can create minimal images that build fast and are super lean. And since we install the same Debian packages in production, creating our images this way also ensures that developers get to work with something that is relatively close to reality.

But Nothing Ever Works Perfectly

Building our images and setting up CI was a pretty straightforward process, but we hit a couple snags while figuring out the user side.

Docker Compose Timing Issues

To make getting the stack up and running as simple as possible, we wanted to use Docker Compose with only a single large docker-compose.yml file that starts and stops everything. Since "everything" consists of more than a dozen containers, this wound up being a little bit tricky to implement.

The issue we ran into is that although Compose is smart enough to start linked containers in the right order, it doesn't have any concept of health checking to determine when the applications within those containers are ready to accept connections.

This can lead to issues like this:

  • Suppose we have 3 containers: Service A, Service B, and Database
  • Service A requires Database to be up and running in order to initialize and start
  • Service B requires Service A to be running in order to initialize and start
  • Now suppose that Database has to run a migration before starting up. Uh oh.

When Compose starts the Database container, the process will start and begin its migration. It will take a minute before the DB is ready to accept connections. However, since Compose doesn't know that, it just moves on to start Service A immediately after the Database instance starts. This results in Service A failing to start, because it can't connect to the DB, which is still running its migration. Service B then tries to come up and fails because Service A hasn't started. Add in a dozen more containers and pretty soon what you've got is a hot mess that is tough to debug.

Luckily, it's easy enough to work around this problem with startup scripts or by using a linked container that blocks until things are ready. This is the path we took (along with changing the init behaviour of some of our services), but it's easy to see things getting complicated if more than just simple health checks are required. Another solution we are thinking about is ditching Compose completely in favour of Ansible. While using Ansible would provide more robust controls, we prefer to stay within the Docker ecosystem if we can.

Connecting From Containers To Localhost

Another issue we ran into was connectivity from containers to services running on localhost. At 500px we mostly use OSX, which means people are running Docker in a VirtualBox VM via Boot2Docker (we haven't tried out Docker Toolbox yet, but definitely plan to).

Although the Boot2Docker OSX installer makes it easy to install VirtualBox and set up inbound port forwarding (that is, connecting from localhost to a container), it is somewhat less obvious how to get outbound connections working from a container to localhost (for example, connecting from inside a container to an API server running locally).

It turns out that it's actually really easy to configure this using PF port forwarding rules, like this:

# In a PF rules file
rdr pass on vboxnet0 inet proto tcp from any to any port 8000 -> port 8000
rdr pass on vboxnet0 inet proto tcp from any to any port 4567 -> port 4567
rdr pass on vboxnet0 inet proto tcp from any to any port 9001 -> port 9001
... etc ...

These rules forward traffic from vboxnet0, the VirtualBox adaptor, to Containers can then access localhost via the IP of the vboxnet0 interface on the OSX side and packets will be routed appropriately (if you are following along, you can find this IP by running boot2docker config | grep HostIP). To persist PF rule changes across reboots, set up a pf.rules file and a pf.plist file as described here and that'll do it.

But this is only really necessary if you actually need to route packets to localhost. We struggled with this for a little bit, but in the end chose an even simpler solution than the one above: we just changed our applications to listen on vboxnet0 instead of lo0. Sometimes, less is more.


So that's where we are so far. We've got containers, CI and a working docker-compose.yml that runs our stack. All we've got to do now is rollout and see what happens. As we start using Docker more and more in our daily workflows, we'll probably run into a few more issues, but we're confident in our technology choices and our ability to work around whatever may come up. We hope this article is interesting and that it is helpful as you plan your own Docker deployments.

Next time around, we'll blog about the things that went right, the things that went wrong, and what we did to make stuff work.

If you have questions, or if you think what we're doing is cool, or even (especially) if you think what we are doing is dumb and batshit insane, drop me a line and let's talk.

As always, we are hiring.