The value of split testing

A while ago we had a discussion over whether we should send our invitation emails as plain text or in HTML format. My gut feeling was that spam filters would be tougher on HTML emails than plain text, but I didn’t know for sure. And I had NO IDEA which email format would have a higher response rate.

Instead of sitting around talking about it — I decided to test it.

We happen to have a conference with several thousand attendees coming up in a few weeks. This gave me a great opportunity to do some split testing with our emails. For the test, I sent out 2459 emails with exactly the same wording. Half of them were in HTML. The other half were in plain text.

Of those 2459 emails, 81 (3.3%) either bounced or were rejected by spam filters. Of the 81 emails that were returned 26 of them were plain text emails, and 55 were HTML emails. This means that the HTML emails were rejected at more than twice the rate of the text emails. However, the response rate to the HTML emails was 11% better than that of the plain text emails!

After taking the lower delivery rate into consideration, the data suggests that I could increase our user participation rate 9% just by sending our emails out in HTML format instead of plain text!

Needless to say, this is an exciting discovery for me. However, I’m not close to done yet. For one, I still need to prove these numbers hold up with other types of conferences. And two, I have a lot of other things to test and refine. I particularly want to see how changing the wording impacts our response rates. I want to test different layouts on our website. What would happen if I changed the background color from blue to green? What if I made the font bigger?

I don’t know — but I’m going to find out!