Continuing where our Part 1 post left off, this post will deal with sending emails in the background.

Sending Email Can Hurt Your App

Suppose you have a user that has just signed up for an account with your company using your shiny new web app. You need to send welcome email outlining some ways to quickly get started with your product and maybe important stuff for future reference. How do you do that?

I bet ninety nine percent of the web apps out there will just send the email in the web request handler. This works fine... most of the time. Now imagine that, with time, the messages that you need to send get larger. Somebody adds a big attachment or switches the content to a Word-generated abomination that superficially looks like HTML. Or maybe a coworker switches the configuration to use a laggy mail server hosted in a different country. Now your server's request queue starts to grow and your other pages get served more slowly. Customers get angry and start turning to your competitors. Managers get angry and developers get stressed out.

Background Sends -- a Possible Solution

So, what is the solution to all those problems? The first thing that crosses people's minds is to move the email sending code out of the web request handler. The usual Django approach would be to use a Celery queue and just fire up a new task that will be processed independently. There are many Django apps that help in doing that. Some of them that I've stumbled upon are:

Those are all fine apps, but I think they may be solving the wrong problem.

Queues before Queues

Now, let me ask you a question: how does an SMTP manage email that it is supposed to send out? That's right -- it already has an internal queue. And that isn't a general queue, but a queue optimized for mail delivery. It makes no sense to me to slap a Celery queue before your mail server queue. Mandatory YO DAWG joke below:

YO DAWG I PUT A QUEUE BEFORE YOUR MAIL QUEUE

So, my solution is to use a real mail server with a real mail queue instead.

SMTP 201: Smarthosts

My solution to the mail queueing problem is to use a local SMTP server configured as a smarthost. A smarthost is a server that will not deliver email directly, but will relay it to another server. With this set up you gain a lot of benefits:

  • You move your mail configuration out of your app. My Django apps don't even have a mail server configuration block since the default localhost endpoint suits me fine. Email configuration now becomes a part of your operations environment (as it really should be).
  • You gain reliability. What happens if your mail server goes offline, or your network connection goes south? Do you lose all those important notifications that absolutely have to reach your customers' inboxes? Your local smarthost will save its mail and will attempt to deliver it later. You even have tools to manage your mail queue, if you want to dig deeper. Compare that to trying to figure out which emails got through and resending the ones that did not.
  • You don't bother with background sends. Local SMTP transactions are fast enough to be done in web requests. Your code will get simpler.

Mechanics: Postfix Configuration

I am using Postfix since I've gotten used to it, but the same technique can be applied to almost any mail server. First you need to install it, which on Debian/Ubuntu systems is a simple:

$ sudo apt-get install postfix

Then you have to edit your /etc/postfix/main.cf file to set up your upstream mail server address and credentials. Here is the config file I am using for one of my servers:

# listen on localhost only and don't expose ourselves to spammers
inet_interfaces = 127.0.0.1

smtp_sasl_auth_enable = yes 
smtp_sasl_password_maps = static:MAILUSER:MAILPASSWORD
smtp_sasl_security_options = noanonymous 
header_size_limit = 4096000
relayhost = [MAILSERVER]:MAILPORT

Make sure you provide the correct values for MAILUSER, MAILPASSWORD, MAILSERVER and MAILPORT and that is all. Note that we tell Postfix to listen to the local interface only, so that we avoid spammers that may find our server.

We also don't enable SSL since all communication happens over the local interface and that saves us some server resources. Speaking of resources, Postfix is pretty lightweight. At the moment, qmgr, the only Postfix process, on one of my servers is running with a resident memory size of 676 KB.

Bonus Points: Use a Mail Delivery Service

I used to be one of the original Postmark developers and I know how hard it is to set up your own mail server. I'd rather have somebody else do all that for me. That is why I recommend that you use Postmark or one of the other delivery services and work on features your app needs instead of fighting mail servers. Oh, and by the way, delivery services usually have tons of other useful features like email history, delivery and bounce reports, incoming mail parsing, etc.