Everything comes down to the IP address

Two of the companies I’ve been most excited about recently are SendGrid and Superfeedr.  These two companies have a lot in common.  For starters, both companies have products aimed squarely at developers like myself.  Both solve real pain points that I’ve experienced firsthand.   Both companies are solving problems that look deceptively simple on the surface.  What do they do?  SendGrid sends emails.  Superfeedr fetches feeds. Simple stuff, right?  Pick any high-level language and you can accomplish either task with 10 lines of code, tops.  And yet, I’m a customer of both of these companies and would invest in either one given the money and the opportunity.

Let’s take them one at a time.

SendGrid

Sending an email is simple.  Getting emails consistently into inboxes is not.  At Eventvue we’ve been plagued with email deliverability issues since our very first conference.  Since sending email is crucial to our business, I invested the time to learn about SPF records, DomainKeys, email whitelists, the spam assassin rules and subscribing to feedback loops.  We did it all.  And yet, we still had problems…

Superfeedr

Fetching a feed is simple.  Fetching thousands of feeds in near real-time is not.  Over the years the code I’ve used to fetch a feed has drastically improved.  I made optimizations to fetch feeds in parallel.  I started keeping track of feeds that failed regularly so I could fix or eliminate them.  I updated my records when feeds were permanently redirected.  I got better at parsing feeds regardless of whether they were RSS or ATOM, version 0.9, 1.0, 2.0 or when they contained invalid XML.  I experimented with predictive algorithms to fetch frequently updated feeds more often.  I wrote a bunch of code but still found myself running into issues when trying to fetch certain feeds…

Okay, but what does this have to do with IP addresses?

Earlier this week Jud Valeski from GNIP spoke at the Boulder CTO lunch.  The topic was “performance and scaling for the real-time web” but the conversation ended up being a lot broader than that.  Jud made one comment in particular that stayed on my mind for the rest of the day.  It was something along the lines of “everything comes down to the IP address.”  It may sound trivial, but that offhand comment was a bit of an “aha” moment for me.

You see, the missing piece for us sending email was the reputation of IPs we were sending emails from. It turns out in the magical equation for getting an email into an inbox, your reputation matters more than anything.  That’s the part where SendGrid can help the most.

And the reason I was having trouble fetching feeds in near real-time?  You guessed it.  My IP address was getting blocked for pounding certain servers too hard.  In contrast, Superfeedr has a much larger pool of IPs to draw from and is in a much better position to deal with those issues.

So everything comes down to the IP — at least when it comes to sending emails and fetching feeds.  The  problem for me as a developer is I don’t want to deal with stuff at that level.  And I’m not alone.  I want to be able to focus on building cool technology instead of worrying about which IP a request is going out on.  This is part of the reason why I’m so bullish about SendGrid and Superfeedr.  For a small monthly fee, they’ll take care of that stuff for me.  Keep in mind that cloud computing is only making these challenges harder.  It’s too easy for someone to fire up an EC2 instance and spam the entire world.  As a result, the entire EC2 IP-block has email reputation issues.  Google App Engine has similar challenges with several services threatening to block App Engine IPs because of abuse by a small number of individuals.  Elastic IPs will reduce this problem, but there’s still the hassle of managing all of that.  For most developers it’s just not worth the time and effort to get things that really ought to be a commodity by now.

So to Julien and Isaac — you’re building stuff that matters.  Keep up the good work.  And Jud, thanks for the spark that got me thinking.

Update: Jud expounded on this thought in a new post titled IP address brokers please stand up

  • http://sendgrid.com Isaac Saldana

    Josh, thanks for the mention. As you have described, the IP reputation of a mail server plays a huge role in ISPs determining whether to send emails to the Inbox or the Junk folder. As a result, we encourage users to get their own IPs so that the good reputation of senders is not shared or affected by the reputation of bad senders. With the wider adoption of DKIM, ISPs will start paying closer attention to the domain reputation but the IP address will always play a role.

  • http://intensedebate.com/people/jud_valeski3968 Jud Valeski

    Eloquent post Josh. Well summarized. I tried to capture more of the thought at http://one.valeski.org/2009/11/ip-address-brokers… , but your post makes the point much better. Psyched this thought is crystallizing.

  • http://superfeedr.com Julien

    Completely agree with that post.

    As you pointed out, us, hackers can always build what we need (and it's the beauty of the internet), however, when we want to "get serious", we have to rely on existing bricks and stop "re-inventing the wheel". To that extent, I think that building "brick" services that can be integrated in any app makes a lot of sense.

    I also fully agree with the important of the IP. Of course, for feeds, (as for emails, I'm sure) there are a lot of other issues, but eventually, the "reputation" of a specific one is key to our business too.

    And of course, thanks a lot Josh for the shout-out and the feedback. We're working hard to have people like you happy, and it's a pleasure!

  • http://twitter.com/JohnBorthwick @JohnBorthwick

    Josh good post. This is exactly why we at betaworks invested in Superfeedr. One of our companies started down the path of building this themselves and we learnt what you outline above — the basic case is simple but at scale its complex. Julien is starting to kick butt at scale. Sounds like SendGrid is as well.

    • http://intensedebate.com/people/joshfraser Josh Fraser

      Thanks for adding your thoughts to this. You made a smart investment.

  • http://blog.timakinbo.com/ Tim Akinbo

    Hey Josh, thanks for this post. I’ve once developed an application for a colleague and sending out huge amounts of email have been a tough challenge. Having implemented SPF records on his domain and throttling of emails, we’ve had our own share of troubles. I’m glad to know that SendGrid does this reliably and that there’s an API.

  • Pingback: If only we lived in an ideal world()