Posts tagged ‘code’


Exporting Robinhood investments to CSV

I recently discovered Robinhood. It’s a mobile trading platform that let’s you invest in the stock market without paying any trading fees. I can’t shake the feeling that 0-fee trading is going to be one of those game-changes that creates a whole range of new applications that weren’t possible before. Quantopian is an early example, but I bet there will be more.

Robinhood is currently only available on mobile and it doesn’t have any of the charts and graphs you would find on any other trading platform. Here’s a little python script I cooked up that will let you export your Robinhood trades to a .csv file you can import into Google Finance or whatever tool you use to track your investments.

Before running this code, make sure you have python & pip installed on your computer. You can either view the code on Github or if you trust me, open your terminal and run the following commands. If you have git installed, you can fetch the files you need using:

git clone https://github.com/joshfraser/robinhood-to-csv.git

Otherwise, manually download the files. Then:

cd robinhood-to-csv
pip install -r requirements.txt
python csv-export.py

The script will request your credentials then prompt you for a filename like this:

Robinhood username:
Password:
1 queued trade and 11 executed trades found in your account.
Choose a filename or press enter to save to `robinhood.csv`:
  comments

Taking a peek inside __VIEWSTATE

If you’ve ever viewed-source on a website that uses Microsoft technology like ASP or .NET, you may have noticed a massive blob of unintelligible text stored in a input field called __VIEWSTATE. What you’re seeing is actually a bunch of Base64 encoded data that gets passed back and forth between the server and the client. I don’t understand how anyone ever thought this was a good idea, but there are a ton of sites that still use this technique. Just check out some old enterprise applications or any Microsoft website and you’ll see what I mean. The United and US Airways websites are a couple other good examples. Unless the __VIEWSTATE is encrypted, you’ll be able to take a look inside using this simple bookmarklet:

javascript:xmp=document.createElement("xmp");
txt=document.createTextNode(atob(document.getElementsByName("__VIEWSTATE")[0].value));
xmp.appendChild(txt);
document.body.insertBefore(xmp,document.body.firstChild);

Drag this link to your toolbar to try it: Decode ViewState

  comments

Update on a few of my open source projects

I’ve been trying to do some cleanup and reorganization of my various open source projects. I regularly get emails asking for help or gently reminding me that one of my projects needs updating. The truth is, most of my projects have been horribly neglected for the last couple years. While I can’t promise my participation will improve, I would like to recognize some of the extraordinary people who have been helping me with maintenance while I’ve been busy. Here’s a quick status update on five of my more popular open source projects.

PubSubHubbub plugin for WordPress

I know, I know, RSS is supposed to be dead. Tell that to the 1,300+ people who downloaded my PubSubHubbub (PuSH) plugin last week, or the 58,000+ people who’ve downloaded it since it was first released. A huge thanks to Matthias Pfefferle for his help maintaining the project. It’s hard to say what’s going to happen to RSS & PuSH, especially as Google kills their Reader product. I expect big companies to continue to push for more centralized systems. Meanwhile, I’ll be here rooting for the open and distributed web however I can.

Timezone detection in JavaScript

My timezone detection article continues to be one of my more popular posts on this blog. Today there are thousands of websites using some variation of my timezone detection code. Jon Nylander took my code and has expanded it into a far more robust solution. His version is way better than mine, so use it instead.

Backwards compatible window.postMessage

My cross-browser implementation of window.postMessage is now pretty stable and is still one of the simplest solutions I’ve seen. I’ve heard that Twitter and Disqus both use easyXDM instead, but I haven’t dug into it myself to know how it compares.

Name Parser

Splitting names is a surprisingly tricky task. My solution isn’t perfect, but it’s a nice step beyond just splitting a name based on a space. Mark Pemburn did a nice job porting my library from PHP to JavaScript. This library has a lot of potential for improvement and I’d love to see more contributions here.

Rolling Curl

Another popular post (especially with the rent-a-coder crowd trying to scrape websites) was my Rolling Curl library in PHP. It solves the blocking connection problem with using cURL in PHP. Alexander Makarow and Fabian Franz have done a great job at improving my code and adding new features.

I’ve found it incredibly rewarding to participate in these projects and others. It blows my mind how much of the world runs on open source software. Open source was one of those things that never fully made sense to me until I started contributing myself. I encourage every developer to find a way to get involved in an open source project and give back. It’s been a great way for me to meet other passionate developers and learn from people who are smarter than I.

  comments

Learn to code

The team at code.org published a terrific video encouraging people to learn to code. It features some of the biggest names in our industry and it does a great job at explaining why programming skills are so important in this day and age. They want every student in every school to have the opportunity to learn to code. I’ve shared before the story of how I became a programmer. I was privileged to have access to a computer at a young age and a dad who encouraged me to learn to code. Considering the ever increasing demand for programming skills, it’s crazy that nine out of ten U.S. schools don’t offer computer programming classes at all. This is a great initiative to raise awareness and help inspire the next generation of developers.

 6 comments

Breadcrumbs in Rails

I’ve been working a little with Ruby on Rails recently.  One of the things I needed was a good module for handling navigational breadcrumbs.  I googled around a bit, but wasn’t able to find anything that fit my needs.  The closest to what I wanted was this example on stackoverflow.  It was a great starting point, but I ran into issues when I wanted to use nested controllers.  I wanted something a little more flexible.  Here’s my final solution.  I’m still a complete noob at rails, so feel free to point out any improvements you may have.

Features:

  • Splits up the URL and looks up the controllers for each section
  • Assumes you have a ‘name’ field on your data structure
  • Doesn’t link the last item since it should be the current pag
  • Works with nested controllers
  • Converts underscores to spaces and titleize’s the labels
  • Easy to modify to fit your own purposes

Code:

def get_bread_crumb(url)
    begin
        breadcrumb = ''
        so_far = '/'
        elements = url.split('/')
        for i in 1...elements.size
       
            so_far += elements[i] + '/'
           
            if elements[i] =~ /^\d+$/
                begin
                    breadcrumb += link_to_if(i != elements.size - 1, eval("#{elements[i - 1].singularize.camelize}.find(#{elements[i]}).name").gsub("_"," ").to_s, so_far)
                rescue
                    breadcrumb += elements[i]
                end
            else
                breadcrumb += link_to_if(i != elements.size - 1,elements[i].gsub("_"," ").titleize, so_far)
            end
           
            breadcrumb += " » " if i != elements.size - 1
        end
        breadcrumb
    rescue
        'Not available'
    end
end

Usage:

<%= get_bread_crumb(request.request_uri) %>

Which converts a URL like this:

/admin/posts/12/comments/8

to breadcrumbs like this:

Admin » Posts » Breadcrumbs in Rails » Comments » Great Post!

 37 comments

Backwards compatible window.postMessage()

You can find the latest version of this library on Github.

Simple cross-domain messaging

This blog post explains how to implement a backwards compatible version of window.postMessage() to handle all your cross-domain messaging needs. If you’re in a hurry, you can skip directly to the demo or just grab the following files:

Background

One of the trickiest things you will ever run into on the web is the same origin policy. The same origin policy basically limits how scripts and frames on different domains can talk to each other.  The same origin policy is an important part of your security on the web. For example, it prevents someone from being able to steal your password from another frame on the page. The annoying thing is there are sometimes perfectly valid reasons for frames on different domains to need to talk to one another.   One good example of this would be the Facebook Connect library where facebook.com needs to be able to communicate with non-facebook.com domains.  Over the years we’ve developed a series of hacks to work around this browser limitation. Some developers have used flash while others have relied on a window.location.hash hack.  Facebook worked around it by getting people to install a cross domain communication channel.  It got pretty ridiculous until the browser makers finally decided to give us a way to do cross-domain messaging without all the nonsense.  The result was window.postMessage() which is supported by the latest browsers like Firefox 3, Safari 4, Chrome and IE 8.  Unfortunately, as usual we’re going to need a backwards compatible version before we can take advantage of this new functionality.

I found a couple great examples of people who have tackled this already.  Luke Shepard wrote xd.js which is part of the open-sourced Facebook Connect code.  I also found Ben Alman’s jQuery plugin which does a really nice job.  Both of these scripts are great, but neither fits quite right with my needs.  For one, I wanted the smallest possible script written in pure JavaScript.  I’m a fan of jQuery, but since I’ll be installing this code on other people’s domains I can’t assume that jQuery will be available and while I could load it up it’s important to keep the file size small.  So what I did was start with Ben’s code and took out all of the jQuery dependencies. Here is the result:

The code

// everything is wrapped in the XD function to reduce namespace collisions
var XD = function(){

    var interval_id,
    last_hash,
    cache_bust = 1,
    attached_callback,
    window = this;

    return {
        postMessage : function(message, target_url, target) {
            if (!target_url) {
                return;
            }
            target = target || parent;  // default to parent
            if (window['postMessage']) {
                // the browser supports window.postMessage, so call it with a targetOrigin
                // set appropriately, based on the target_url parameter.
                target['postMessage'](message, target_url.replace( /([^:]+:\/\/[^\/]+).*/, '$1'));
            } else if (target_url) {
                // the browser does not support window.postMessage, so use the window.location.hash fragment hack
                target.location = target_url.replace(/#.*$/, '') + '#' + (+new Date) + (cache_bust++) + '&' + message;
            }
        },
        receiveMessage : function(callback, source_origin) {
            // browser supports window.postMessage
            if (window['postMessage']) {
                // bind the callback to the actual event associated with window.postMessage
                if (callback) {
                    attached_callback = function(e) {
                        if ((typeof source_origin === 'string' && e.origin !== source_origin)
                        || (Object.prototype.toString.call(source_origin) === "[object Function]" && source_origin(e.origin) === !1)) {
                             return !1;
                         }
                         callback(e);
                     };
                 }
                 if (window['addEventListener']) {
                     window[callback ? 'addEventListener' : 'removeEventListener']('message', attached_callback, !1);
                 } else {
                     window[callback ? 'attachEvent' : 'detachEvent']('onmessage', attached_callback);
                 }
             } else {
                 // a polling loop is started & callback is called whenever the location.hash changes
                 interval_id && clearInterval(interval_id);
                 interval_id = null;
                 if (callback) {
                     interval_id = setInterval(function() {
                         var hash = document.location.hash,
                         re = /^#?\d+&/;
                         if (hash !== last_hash && re.test(hash)) {
                             last_hash = hash;
                             callback({data: hash.replace(re, '')});
                         }
                     }, 100);
                 }
             }
         }
    };
}();

Usage:

There are two parts to using this code: posting and listening. Both are relatively simple. To post a message we call XD.postMessage with a message, a URL and the frame that we want to talk to. Notice that we start off by passing the URL of the parent page to the child frame. This is important so the child knows how to talk back to the parent.

// pass the URL of the current parent page to the iframe using location.hash
src = 'http://joshfraser.com/code/postmessage/child.html#' + encodeURIComponent(document.location.href);
document.getElementById("xd_frame").src = src;

function send(msg) {
    XD.postMessage(msg, src, frames[0]);
    return false;
}

Setting up the listener on the child is also easy to do:

var parent_url = decodeURIComponent(document.location.hash.replace(/^#/, ''));

XD.receiveMessage(function(message){
    window.alert(message.data + " received on "+window.location.host);
}, 'http://onlineaspect.com');

I recommend taking a look at this barebones example to understand better how the various pieces fit together. This is still a work in progress and I’d love any feedback you have on it. I’m particularly interested in adding Flash as an alternative method before falling back to fragments. This is what the Facebook code does and I like it because it eliminates the nasty polling every 100ms.

Got other thoughts on how to make this better? Let me know in the comments.

 55 comments

Splitting names

You can find the latest version of this code on Github. There are libraries for both PHP and JavaScript.

The quest

I’m on a ongoing search to find the best algorithm for splitting a full name into a first name and a last name. I’m sure this sounds like a ridiculously trivial quest — just explode the string on a space, right?

The challenge

But how do you tell the difference between people with double first names like Jo Ann Smith and people with double last names like Jo Von Trapp? What would you do if I gave you a double first name AND and a double last name at the same time?

Did you remember that you might need to parse out prefixes (Mr, Mrs, etc) and suffixes (II, Jr, PhD, etc)?

How do you turn Paul T. S. Williams into Paul Williams while intelligently deducing that that T. James Adams probably wants to go by James Adams, but T. Adams should probably stay as T. Adams?

And how do you straighten out the capitalization? I MIGHT WRITE IN ALL CAPS or all lowercase. Most names have the first letter capitalized and everything else in lowercase, but of course there are exceptions. J.P. likes to have both initials capitalized and Mr. McDonald always gets fussy when you forget to capitalize the D. Oh, and I hope you’re prepared for other anomalies like people with dashes in their name.

As you have probably realized by now, splitting a full name into its proper parts is a little more complicated than it appears on the surface.

I wrote the first version of my name-parsing algorithm two years ago and I’ve been gradually refining it ever since. It’s not perfect, but it’s improved a lot over time. I’m posting this code along with a demo in hopes that it will spur contributions to improve its accuracy even more. Throw the hardest names you know at it and let me know how it performs. I know I’m missing words for the various dictionaries of prefixes, suffixes, and compound name identifiers. Please let me know what I missed.

The algorithm

We start by splitting the full name into separate words. We then do a dictionary lookup on the first and last words to see if they are a common prefix or suffix. Next, we take the middle portion of the string (everything minus the prefix & suffix) and look at everything except the last word of that string. We then loop through each of those words concatenating them together to make up the first name. While we’re doing that, we watch for any indication of a compound last name. It turns out that almost every compound last name starts with 1 of 15 prefixes (Von, Van, Vere, etc). If we see one of those prefixes, we break out of the first name loop and move on to concatenating the last name. We handle the capitalization issue by checking for camel-case before uppercasing the first letter of each word and lowercasing everything else. I wrote special cases for periods and dashes. We also have a couple other special cases, like ignoring words in parentheses all-together.

The code

<?

// split full names into the following parts:
// - prefix / salutation  (Mr., Mrs., etc)
// - given name / first name
// - middle initials
// - surname / last name
// - suffix (II, Phd, Jr, etc)
function split_full_name($full_name) {
    $full_name = trim($full_name);
    // split into words
    $unfiltered_name_parts = explode(" ",$full_name);
    // completely ignore any words in parentheses
    foreach ($unfiltered_name_parts as $word) {
        if ($word{0} != "(")
            $name_parts[] = $word;
    }
    $num_words = sizeof($name_parts);

    // is the first word a title? (Mr. Mrs, etc)
    $salutation = is_salutation($name_parts[0]);
    $suffix = is_suffix($name_parts[sizeof($name_parts)-1]);

    // set the range for the middle part of the name (trim prefixes & suffixes)
    $start = ($salutation) ? 1 : 0;
    $end = ($suffix) ? $num_words-1 : $num_words;

    // concat the first name
    for ($i=$start; $i < $end-1; $i++) {
        $word = $name_parts[$i];
        // move on to parsing the last name if we find an indicator of a compound last name (Von, Van, etc)
        // we use $i != $start to allow for rare cases where an indicator is actually the first name (like "Von Fabella")
        if (is_compound_lname($word) && $i != $start)
            break;
        // is it a middle initial or part of their first name?
        // if we start off with an initial, we'll call it the first name
        if (is_initial($word)) {
            // is the initial the first word?  
            if ($i == $start) {
                // if so, do a look-ahead to see if they go by their middle name
                // for ex: "R. Jason Smith" => "Jason Smith" & "R." is stored as an initial
                // but "R. J. Smith" => "R. Smith" and "J." is stored as an initial
                if (is_initial($name_parts[$i+1]))
                    $fname .= " ".strtoupper($word);
                else
                    $initials .= " ".strtoupper($word);
            // otherwise, just go ahead and save the initial
            } else {
                $initials .= " ".strtoupper($word);
            }
        } else {
            $fname .= " ".fix_case($word);
        }  
    }

    // check that we have more than 1 word in our string
    if ($end-$start > 1) {
        // concat the last name
        for ($i; $i < $end; $i++) {
            $lname .= " ".fix_case($name_parts[$i]);
        }
    } else {
        // otherwise, single word strings are assumed to be first names
        $fname = fix_case($name_parts[$i]);
    }

    // return the various parts in an array
    $name['salutation'] = $salutation;
    $name['fname'] = trim($fname);
    $name['initials'] = trim($initials);
    $name['lname'] = trim($lname);
    $name['suffix'] = $suffix;
    return $name;
}

// detect and format standard salutations
// I'm only considering english honorifics for now & not words like
function is_salutation($word) {
    // ignore periods
    $word = str_replace('.','',strtolower($word));
    // returns normalized values
    if ($word == "mr" || $word == "master" || $word == "mister")
        return "Mr.";
    else if ($word == "mrs")
        return "Mrs.";
    else if ($word == "miss" || $word == "ms")
        return "Ms.";
    else if ($word == "dr")
        return "Dr.";
    else if ($word == "rev")
        return "Rev.";
    else if ($word == "fr")
        return "Fr.";
    else
        return false;
}

//  detect and format common suffixes
function is_suffix($word) {
    // ignore periods
    $word = str_replace('.','',$word);
    // these are some common suffixes - what am I missing?
    $suffix_array = array('I','II','III','IV','V','Senior','Junior','Jr','Sr','PhD','APR','RPh','PE','MD','MA','DMD','CME');
    foreach ($suffix_array as $suffix) {
        if (strtolower($suffix) == strtolower($word))
            return $suffix;
    }
    return false;
}

// detect compound last names like "Von Fange"
function is_compound_lname($word) {
    $word = strtolower($word);
    // these are some common prefixes that identify a compound last names - what am I missing?
    $words = array('vere','von','van','de','del','della','di','da','pietro','vanden','du','st.','st','la','ter');
    return array_search($word,$words);
}

// single letter, possibly followed by a period
function is_initial($word) {
    return ((strlen($word) == 1) || (strlen($word) == 2 && $word{1} == "."));
}

// detect mixed case words like "McDonald"
// returns false if the string is all one case
function is_camel_case($word) {
    if (preg_match("|[A-Z]+|s", $word) && preg_match("|[a-z]+|s", $word))
        return true;
    return false;
}

// ucfirst words split by dashes or periods
// ucfirst all upper/lower strings, but leave camelcase words alone
function fix_case($word) {
    // uppercase words split by dashes, like "Kimura-Fay"
    $word = safe_ucfirst("-",$word);
    // uppercase words split by periods, like "J.P."
    $word = safe_ucfirst(".",$word);
    return $word;
}

// helper function for fix_case
function safe_ucfirst($seperator, $word) {
    // uppercase words split by the seperator (ex. dashes or periods)
    $parts = explode($seperator,$word);
    foreach ($parts as $word) {
        $words[] = (is_camel_case($word)) ? $word : ucfirst(strtolower($word));
    }
    return implode($seperator,$words);
}

?>
 6 comments

Having fun with Proximity for mac

One of the things I love about my mac is how easy it is to hack things to work the way I want.  I’m always amazed by how many easy hooks there are into system settings and native applications.

I recently stumbled upon a neat application called Proximity. Proximity detects when a selected device (cell phone, wireless mouse, etc) comes in or out of bluetooth range and executes selected scripts. Since my iPhone is almost always with me, I decided to write a couple scripts to password protect my laptop when my iPhone isn’t around, and unlock it when I return. As an added bonus, my code also mutes my audio and sets an away message on iChat when I leave. It then sets my status to “available” when I return.

The cool thing about this is that it keeps my laptop secure without having to mess with a screen-saver password all the time. I can think of a lot of other uses for this technology. For example, I wonder how many people would like to have a notification pop up when their boss is about to walk into the room, or just have a bluetooth device automatically sync when it’s in range of their computer. I should add that Bluetooth detection has its limitations, particularly because the underlying hardware makes it tough to detect realtime changes causing a significant lag. You also don’t have anyway to detect the strength of the signal to get any sense of how far away the device is from your computer — it’s entirely binary — the device is on and in-range or it’s not. That said, it’s still a powerful demonstration of what can be accomplished with technology when you start getting creative.

Here are my scripts. First, the one that gets executed whenever my iPhone goes out of range:

-- mute volume
set volume with output muted

-- set status to away
tell application "iChat"
    set status to away
end tell

-- turn on the screen saver password
tell application "System Events"
    tell security preferences
        set properties to {require password to wake:true}
    end tell
end tell

-- activate the screen saver
tell application "ScreenSaverEngine" to activate

-- if the above line doesn't work, try uncommenting this instead:
-- do script "/System/library/Frameworks/Screensaver.framework/Resources/ScreenSaverEngine.app/Contents/MacOS/ScreenSaverEngine"

And, in range:

-- set status to available
tell application "iChat"
    set status to available
end tell

-- disable screen saver password
tell application "System Events"
    tell security preferences
        set properties to {require password to wake:false}
    end tell
end tell

-- turn off the screen saver
tell application "ScreenSaverEngine" to quit

Let me know if you come up with any other applications for this or have suggestions for other functionality I should add to my fancy phone-triggered security system.

 15 comments

Queue events that occur before JavaScript is loaded

One of the common recommendations for speeding up your website is to put your JavaScript at the bottom of your page instead of including it inside the head tag. The difference this simple placement can have is impressive, especially if you are dealing with sizable JavaScript libraries that are usually 50k at best.

One downside with putting your JavaScript at the bottom is that your fast clicking visitors may click on links that won’t work. The reason this happens is because the JavaScript that those links trigger hasn’t been downloaded yet. Usually those links will work on the second or third try, but it makes for a bad user experience and a poor first impression.

I decided to fix it by queuing up those user-triggered actions and replaying them as soon as the document is ready. I wrote a wrapper that I can use anytime I have code that depends on my JavaScript being downloaded and available.

The concept is simple. Instead of calling functions directly when a user triggers an action, I add the function call to a queue. When the document is ready, I loop through the queue and execute each of the actions in the order that they occurred. I put this code inline inside my head tag so it is available as soon possible. The rest of my JavaScript can then be included right before the closing body tag without worrying about this race condition between the browser and website visitor.

<script type="text/javascript">

var loaded = false;
var action_queue = new Array();

function when_ready(callback) {
    // skip the queue if the document has already loaded
    if (loaded == true)
        eval(callback);
    else {
        action_queue.push(callback);
    }
}

function dequeue_actions() {
    for (i in action_queue) {
        eval(action_queue[i]);
        delete(action_queue[i]); // cleanup after ourselves
    }
    loaded = true;
}

</script>

I then trigger dequeue_actions() as soon as the document is ready:

// using jQuery
$(document).ready(function(){
    dequeue_actions();
});

// this works too
onload = dequeue_actions;

You can then safely make function calls using when_ready(). For example:

<a onclick="select('foo')">foo</a>

becomes

<a onclick="when_ready('select(\'foo\')')">foo</a>

In my testing, the results have been very smooth with delays being almost unnoticeable. Of course, your experience will vary depending on the size of your document and how long it takes for your document to be ready.

This code is pure JavaScript and should work in every modern browser. I’ve tested it in IE6+, FF2+ and Safari 3+.

 9 comments

Reading GET variables with JavaScript

One of the things that isn’t immediately obvious in JavaScript is how to access GET variables. I’ve seen lots of different implementations for this around the web, but the majority of them are bulkier than they need to be. Here’s my favorite way to do it:

<script type="text/javascript">
    function $_GET(q,s) {
        s = s ? s : window.location.search;
        var re = new RegExp('&'+q+'(?:=([^&]*))?(?=&|$)','i');
        return (s=s.replace(/^?/,'&').match(re)) ? (typeof s[1] == 'undefined' ? '' : decodeURIComponent(s[1])) : undefined;
    }
</script>

What this gives you is a JavaScript implementation of PHP’s $_GET functionality.  I use a regular expression to keep the code to a minimum. Here is a simple example of how to use it:

// this code would print "hello world" if it was at http://localhost/index.php?var1=hello&var2=world
var var1 = $_GET('var1');
var var2 = $_GET('var2');
document.write(var1 + " " + var2);

Another thing I like about this implementation is that it makes it easy to parse GET variables from arbitrary search strings (ex “?var1=hello&var2=world”).  This is handy if you need to access GET variables from an HTML src parameter such as an image or script tag.

// get the src parameter and split it down to the search query string
var src = document.getElementById('example').src;
params = src.split('?');
var var1 = $_GET('var1','?'+params[1]);
Updated 01/14/11 with Kip Robinson’s improvements from the comments
 34 comments