Posts tagged ‘ips’

If only we lived in an ideal world

After my last post on the importance of IP addresses, I emailed a few friends I respect for feedback.  I’m glad I did because there’s an important aspect I neglected to address.  Brett Slatkin made this comment:

The post makes sense. What I’d focus on in part is that reputation based on IP is fundamentally broken, which is why email sending with SMTP and feed fetching through polling are both fundamentally flawed. We need better protocols that use client tokens (like OAuth) to do rate limiting instead of using just IP addresses.

I hope this would be obvious, but neither SendGrid nor SuperFeedr are optimal solutions.  Let me be more blunt.  In an ideal world, neither company would have reason to exist.

I also understand that as an entrepreneur you have to dream big while also realizing that significant change usually doesn’t happen overnight.  One day we may all drive Tesla’s, but that doesn’t mean we can do away with our gas stations quite yet.

I’m a firm believer that companies should focus on solving the problems that exist today while being ready to adapt as better solutions get adopted.  For the sake of completeness, here’s the missing footnote to my last post:

The ideal solution to the problem of polling feeds is PubSubHubbub.  The ideal solution for accessing authenticated APIs would be for parties to use token based authentication like OAuth or WRAP.  And as Isaac Saldana mentioned in the comments, the ideal solution for email reputation would be widespread adoption of DKIM which moves the reputation to the domain instead of the IP address.

Those are the ideal solutions.  Too bad we don’t live in an ideal world.


Everything comes down to the IP address

Two of the companies I’ve been most excited about recently are SendGrid and Superfeedr.  These two companies have a lot in common.  For starters, both companies have products aimed squarely at developers like myself.  Both solve real pain points that I’ve experienced firsthand.   Both companies are solving problems that look deceptively simple on the surface.  What do they do?  SendGrid sends emails.  Superfeedr fetches feeds. Simple stuff, right?  Pick any high-level language and you can accomplish either task with 10 lines of code, tops.  And yet, I’m a customer of both of these companies and would invest in either one given the money and the opportunity.

Let’s take them one at a time.


Sending an email is simple.  Getting emails consistently into inboxes is not.  At Eventvue we’ve been plagued with email deliverability issues since our very first conference.  Since sending email is crucial to our business, I invested the time to learn about SPF records, DomainKeys, email whitelists, the spam assassin rules and subscribing to feedback loops.  We did it all.  And yet, we still had problems…


Fetching a feed is simple.  Fetching thousands of feeds in near real-time is not.  Over the years the code I’ve used to fetch a feed has drastically improved.  I made optimizations to fetch feeds in parallel.  I started keeping track of feeds that failed regularly so I could fix or eliminate them.  I updated my records when feeds were permanently redirected.  I got better at parsing feeds regardless of whether they were RSS or ATOM, version 0.9, 1.0, 2.0 or when they contained invalid XML.  I experimented with predictive algorithms to fetch frequently updated feeds more often.  I wrote a bunch of code but still found myself running into issues when trying to fetch certain feeds…

Okay, but what does this have to do with IP addresses?

Earlier this week Jud Valeski from GNIP spoke at the Boulder CTO lunch.  The topic was “performance and scaling for the real-time web” but the conversation ended up being a lot broader than that.  Jud made one comment in particular that stayed on my mind for the rest of the day.  It was something along the lines of “everything comes down to the IP address.”  It may sound trivial, but that offhand comment was a bit of an “aha” moment for me.

You see, the missing piece for us sending email was the reputation of IPs we were sending emails from. It turns out in the magical equation for getting an email into an inbox, your reputation matters more than anything.  That’s the part where SendGrid can help the most.

And the reason I was having trouble fetching feeds in near real-time?  You guessed it.  My IP address was getting blocked for pounding certain servers too hard.  In contrast, Superfeedr has a much larger pool of IPs to draw from and is in a much better position to deal with those issues.

So everything comes down to the IP — at least when it comes to sending emails and fetching feeds.  The  problem for me as a developer is I don’t want to deal with stuff at that level.  And I’m not alone.  I want to be able to focus on building cool technology instead of worrying about which IP a request is going out on.  This is part of the reason why I’m so bullish about SendGrid and Superfeedr.  For a small monthly fee, they’ll take care of that stuff for me.  Keep in mind that cloud computing is only making these challenges harder.  It’s too easy for someone to fire up an EC2 instance and spam the entire world.  As a result, the entire EC2 IP-block has email reputation issues.  Google App Engine has similar challenges with several services threatening to block App Engine IPs because of abuse by a small number of individuals.  Elastic IPs will reduce this problem, but there’s still the hassle of managing all of that.  For most developers it’s just not worth the time and effort to get things that really ought to be a commodity by now.

So to Julien and Isaac — you’re building stuff that matters.  Keep up the good work.  And Jud, thanks for the spark that got me thinking.

Update: Jud expounded on this thought in a new post titled IP address brokers please stand up