Posts tagged ‘email deliverability’

Too many DNS lookups in an SPF record

I recently noticed I was having new email deliverability issues. It surprised me since things had been going well since switching to AuthSMTP for our outgoing mail. The first thing I checked was my SPF record. It looked like this:

v=spf1 a mx -all

At first glance everything seems okay. Basically it says to include all A records, MX records, and to include the SPF records provided by Google Apps, AuthSMTP and Salesforce. Since that covers every legitimate sender, I finish it off with the -all which indicates a hard fail. Ok, so the syntax is good. You can’t tell that anything is wrong without digging a little deeper. When you actually try to evaluate it you’ll get this error message:

Results – PermError SPF Permanent Error: Too many DNS lookups

After a little research I found out that you are only allowed 10 DNS lookups and fetching the TXT and SPF records count toward that total. That means after you add in the A and MX lookups, we’re at 7 before we even look inside the includes. Let’s pull up the SPF record for Google Apps:


That redirect counts as another DNS lookup. That puts me up to 8 DNS lookups. Thankfully the Salesforce SPF record is nice and clean:

v=spf1 ip4: ip4: ip4: ip4: ip4: mx ~all

That leaves AuthSMTP:

v=spf1 ~all

Ouch! That’s 4 more lookups and the worst part of it is that doesn’t even do anything!

The first thing I did was take out the MX lookup since it’s redundant. I also replaced with which is what it redirects to anyway. Technically, this isn’t a good idea since Google could change it on me — but remember I don’t have a lot of options here. I’m just happy to see my revised record pass the test:

v=spf1 a -all

I also sent an email to the AuthSMTP team. They responded within 30 minutes saying that they would remove the extra DNS record and look at how they can clean things up.

I learned something tonight. Remember to count the DNS lookups in your SPF record. It turns out they can add up faster than points on a teenagers drivers license. And if you’re using a lot of includes like I am, remember to do periodic checks to make sure nothing has changed.


  • I wrote about Sending email through Gmail over a year ago. While I absolutely don’t recommend you try this anymore, it has some useful information on SPF records and email deliverability in general.
  • Kitterman have a great tool to help validate your SPF records.
Nowadays I recommend everyone use SendGrid for sending email

3 ways to guard your email reputation

At EventVue, our first point of contact with our users is by email. That’s why I’m obsessive about making a great first impression with our emails. It’s also the reason I work so hard to make sure that we maintain high deliverability rates.

According to George Bilbrey at Return Path, email deliverability is all about sender reputation. When people hit the “mark as spam” button, it hurts your reputation. When they click “not spam”, it helps. When you send hundreds of emails to addresses that don’t exist, your might as well tattoo “SPAMMER” onto your forehead!

What do you do when your email list is made up of unverified email addresses?

How do you tell which ones are good and which ones are bad? More importantly, what can you do to reduce the number of bounced messages and the inevitable damage to your sender reputation? Here are a few things that will help:

1) First pass: regular expressions

Your first defense in checking you have a valid email address is making sure it looks right. Pete Warden recently posted some great examples of how to use regular expressions. In PHP, you can use the preg_match() function:

$exp = '/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[a-zA-Z]{2,4}/';
if (preg_match($exp, $email) <= 0)
    echo "bad email";

2) Take it to the next level: check for an MX record

A lot of people run their email addresses through a regular expression and quit there. I’ve found that doing a quick DNS lookup goes a long way towards eliminating typos from dyslexic users. In PHP, it’s easy to check whether or not a domain has a DNS record for a mail server:

function mx_record_exists($email) {
    $email_parts = split("@", $email);
    // contains exactly 1 @ sign
    if (sizeof($email_parts) != 2)
        return false;

    list($username, $domain) = $email_parts;
    if (checkdnsrr($domain, "MX")) {
        return true;
    else {
        return false;

3) Learn from your mistakes: watch your feedback loops

If an invalid email slips past your first two defenses, it doesn’t mean you have to make that mistake again. Create a blacklist for any bounced emails and make sure you don’t send an email to that address again.


The value of split testing

A while ago we had a discussion over whether we should send our invitation emails as plain text or in HTML format. My gut feeling was that spam filters would be tougher on HTML emails than plain text, but I didn’t know for sure. And I had NO IDEA which email format would have a higher response rate.

Instead of sitting around talking about it — I decided to test it.

We happen to have a conference with several thousand attendees coming up in a few weeks. This gave me a great opportunity to do some split testing with our emails. For the test, I sent out 2459 emails with exactly the same wording. Half of them were in HTML. The other half were in plain text.

Of those 2459 emails, 81 (3.3%) either bounced or were rejected by spam filters. Of the 81 emails that were returned 26 of them were plain text emails, and 55 were HTML emails. This means that the HTML emails were rejected at more than twice the rate of the text emails. However, the response rate to the HTML emails was 11% better than that of the plain text emails!

After taking the lower delivery rate into consideration, the data suggests that I could increase our user participation rate 9% just by sending our emails out in HTML format instead of plain text!

Needless to say, this is an exciting discovery for me. However, I’m not close to done yet. For one, I still need to prove these numbers hold up with other types of conferences. And two, I have a lot of other things to test and refine. I particularly want to see how changing the wording impacts our response rates. I want to test different layouts on our website. What would happen if I changed the background color from blue to green? What if I made the font bigger?

I don’t know — but I’m going to find out!

 1 comment

Let’s start a new trend

This mailbox is not monitored and you will not receive a response.

Every email I receive these days seems to contain that sentence in one form or other. A bit rude, isn’t it? Wait, it gets better…

To ensure delivery, please add us to your address book.

Let me get this straight — you want me to add you to my address book, but you won’t even read my emails? Instead you’re going to make me take a 20 minute scavenger hunt around your website to find out how to get in touch with you?


I’d rather just hit reply.

Here’s an idea: Why not start listening to your customers instead of insisting on having a one-way conversation all the time?

It’s easy:

  • Set up a filter to delete all the “Out of Office” emails and the delayed delivery notices.
  • Forward the bounced emails to your email management script to remove them from your mailing list.
  • Actually start reading and responding to what your users/customers have to say.

Let’s start a new trend and stop using that annoying “no-reply” email address. Come on — we can be more creative than that!


Tutorial: Sending email through Gmail

Nowadays I recommend everyone use SendGrid for sending email

An introduction to using Gmail as your Email Service Provider
There are many reasons to consider outsourcing your email. Getting an email into someone’s inbox is not a simple matter these days. There are a million things you need to know to make sure an email actually makes it to the recipient and not into the junk folder. That’s why more and more people are opting to have an email service provider (ESP) take care of sending email for them. With an ESP, you let them keep up with all the blacklists and the whitelists. You can stop worrying about maintaining mail servers and start focusing on the real challenges of your business.

There are a few downsides to using an ESP. First, they are relatively expensive. For a good ESP, you are looking at paying thousands of dollars a year just to send a few emails. Secondly, most ESP’s are focused towards corporate bulk mailers instead of web startups that send lots of customized emails to one person at a time. This makes integration really awkward. Lastly, the ESP’s that have API’s don’t make it easy on you. It’s hard to find an ESP that will integrate nicely with your existing code.

Recently I decided to try using Google Apps for your Domain to send emails from our website. Google Apps offers many of the same benefits as an ESP, but they do it for free and code integration is a non-issue. So far, the results have been great and we’re not losing as many emails to spam filters anymore. I’m not an deliverability expert, but I learned a lot from fighting the email battle this past week. My goal with this tutorial is to share what I learned, so you don’t have to spend a week running all over the web to find it.

The step-by-step tutorial I wish I’d had a week ago:

Set up Google Apps for your domain
Google Apps for your domain comes in a free Standard Edition and a $50/user/year Premier Edition. I took advantage of their offer of one month free with the Premier Edition (mainly for their included tech support), but ultimately I decided that the Standard Edition more than meets my needs. Unless you’ve got some crazy integration needs, Standard Edition will probably do just fine.

Verify your domain
Verifying your domain allows GMail to send emails from your domain instead of on behalf of your domain. You can verify your domain name by either uploading a specific HTML file to your site, or by adding a CNAME record to your DNS records.

Create an account for each email address you want to send from
Google makes it easy to set up multiple accounts with an Excel spreadsheet upload. I’d recommend using the same password for each account to simplify your life and code. If you’re like me, the first thing you will want to do is sign in to each account and forward everything to an email address that you check every day.

Add Google’s MX servers to your DNS records
You will need to add these MX records for Google’s mail servers to your DNS records

Publish an SPF record that includes Google’s SPF record.
A lot of spam filters rely on the Sender Policy Framework (SPF) and Sender ID to decide whether or not to allow your message through or not. An SPF record is simply a DNS record on your server that has a list of IP addresses and domain names that are allowed to send emails from your domain name. In addition to listing the IP addresses of your own servers, you can also include the SPF record from another domain. In this case, we want to include Google’s SPF record to include an up to date list of their mail servers. If you are going to be sending mail from other servers besides google, I recommend you look over this list of common mistakes before you start playing around with your SPF record. You should be particularly careful if you have existing servers that are sending out mail. It’s easy to lock them out with a bad SPF record.

Use a hard fail (for Hotmail’s behalf)
Your SPF record should end with either “~all” or “-all”. The “~all” is a soft fail. This means that servers not listed in your SPF record should be given extra scrutiny. The “-all” is a hard fail. This means that any server that tries to send an email from your domain that is not included on your SPF record will be rejected. Google recommends you use a soft fail (~all) to ensure deliverability, but in practice I found that specifying a hard fail was the only way to get an email into a Hotmail inbox. The hard fail communicates your confidence that your email system is secure. Most spam filters like that.

Submit your SPF record to Hotmail
You need to submit your SPF record to Hotmail to be included in their Sender ID program. Please note that Hotmail caches their records daily. This means if you change something you will need to resubmit your record in order to have your changes take effect immediately.

Subscribe to feedback loops
Feedback loops provide a way for ISP’s to let email senders know when people are clicking the “report spam” button. Google actively subscribes to these feedback loops so they can stop people from sending spam from their mail servers. Google allows you to be informed about these complaints as well. This allows you to quickly remove the complainers from your mailing list. You can get a copy of these complaints by creating email lists for and

Make sure you include a return-path in your email header
Sender ID checks are performed against the purported responsible address (PRA). SPF checks are performed against the return-path (or bounce address). This means you need to make sure you are sending the correct return-path in your email header and not something generic like “noreply@localhost”.

Make sure you are sending quality emails
There are a lot of things you can do to the emails themselves to increase your chances of getting past spam filters. There are a couple “no-brainers”, like making sure you include an unsubscribe link and use valid HTML. There are also a few not-so-obvious things you can do like adding your physical mailing address to the footer of each email. It’s worth taking some time to research what spam filters care about so you can modify your emails accordingly.

Additional Resources:

Please chip in your thoughts / questions. Good luck!