Everything comes down to the IP address

by on November 20, 2009


Two of the companies I’ve been most excited about recently are SendGrid and Superfeedr.  These two companies have a lot in common.  For starters, both companies have products aimed squarely at developers like myself.  Both solve real pain points that I’ve experienced firsthand.   Both companies are solving problems that look deceptively simple on the surface.  What do they do?  SendGrid sends emails.  Superfeedr fetches feeds. Simple stuff, right?  Pick any high-level language and you can accomplish either task with 10 lines of code, tops.  And yet, I’m a customer of both of these companies and would invest in either one given the money and the opportunity.

Let’s take them one at a time.

SendGrid

Sending an email is simple.  Getting emails consistently into inboxes is not.  At Eventvue we’ve been plagued with email deliverability issues since our very first conference.  Since sending email is crucial to our business, I invested the time to learn about SPF records, DomainKeys, email whitelists, the spam assassin rules and subscribing to feedback loops.  We did it all.  And yet, we still had problems…

Superfeedr

Fetching a feed is simple.  Fetching thousands of feeds in near real-time is not.  Over the years the code I’ve used to fetch a feed has drastically improved.  I made optimizations to fetch feeds in parallel.  I started keeping track of feeds that failed regularly so I could fix or eliminate them.  I updated my records when feeds were permanently redirected.  I got better at parsing feeds regardless of whether they were RSS or ATOM, version 0.9, 1.0, 2.0 or when they contained invalid XML.  I experimented with predictive algorithms to fetch frequently updated feeds more often.  I wrote a bunch of code but still found myself running into issues when trying to fetch certain feeds…

Okay, but what does this have to do with IP addresses?

Earlier this week Jud Valeski from GNIP spoke at the Boulder CTO lunch.  The topic was “performance and scaling for the real-time web” but the conversation ended up being a lot broader than that.  Jud made one comment in particular that stayed on my mind for the rest of the day.  It was something along the lines of “everything comes down to the IP address.”  It may sound trivial, but that offhand comment was a bit of an “aha” moment for me.

You see, the missing piece for us sending email was the reputation of IPs we were sending emails from. It turns out in the magical equation for getting an email into an inbox, your reputation matters more than anything.  That’s the part where SendGrid can help the most.

And the reason I was having trouble fetching feeds in near real-time?  You guessed it.  My IP address was getting blocked for pounding certain servers too hard.  In contrast, Superfeedr has a much larger pool of IPs to draw from and is in a much better position to deal with those issues.

So everything comes down to the IP — at least when it comes to sending emails and fetching feeds.  The  problem for me as a developer is I don’t want to deal with stuff at that level.  And I’m not alone.  I want to be able to focus on building cool technology instead of worrying about which IP a request is going out on.  This is part of the reason why I’m so bullish about SendGrid and Superfeedr.  For a small monthly fee, they’ll take care of that stuff for me.  Keep in mind that cloud computing is only making these challenges harder.  It’s too easy for someone to fire up an EC2 instance and spam the entire world.  As a result, the entire EC2 IP-block has email reputation issues.  Google App Engine has similar challenges with several services threatening to block App Engine IPs because of abuse by a small number of individuals.  Elastic IPs will reduce this problem, but there’s still the hassle of managing all of that.  For most developers it’s just not worth the time and effort to get things that really ought to be a commodity by now.

So to Julien and Isaac — you’re building stuff that matters.  Keep up the good work.  And Jud, thanks for the spark that got me thinking.

Update: Jud expounded on this thought in a new post titled IP address brokers please stand up