Salmon Run: Searchmash Javascript client using Prototype

Saturday, June 21, 2008

Searchmash Javascript client using Prototype

I haven't used Javascript for a while. The last time I used it actively, to consume JSON results (generated from local backend components) on a web page, was over three years ago, and even then, I would deliberately keep the Javascript side real simple, doing all the processing of the JSON in a server component and then just popping the formatted HTML output into the innerHTML of the div element on the web page. In my defense, this was before all these Javascript frameworks that wrap the XmlHttpRequest up into nice functions, and decent Javascript debuggers such as Firebug. So the Javascript was complicated enough without having to compose HTML from JSON at the browser side.

Lately, however, I have been thinking of ways clients can leverage our API (which returns RSS 2.0 XML results by default, but can return JSON results if requested with output=json on the query parameters). During the last two years, Javascript has become more popular, various frameworks have matured and debuggers have improved. So trying these tools out and getting a feel for them tools would not only update my skills to something approaching real-world Javascript programmers, but also allow me to apply the lessons learnt here, so I can advise clients on how they can use our API in different ways.

After I moved out of the Javascript-heavy project I mentioned earlier, others in our group continued to improve the application, and I kept hearing real good things about this (then new) Javascript framework called Prototype, which provided a nice set of functions that made Javascript coding easier and much more fun. So I decided to try out Prototype first, in order to make a Javascript based widget to return search results for my blog, using Searchmash (the apparently secret Google JSON API) as the search results provider.

The first problem I ran into was Javascript's same origin policy restriction. According to this, Javascript would not allow me to make calls on a remote server. The workaround for this is to set up a proxy on your own site that will forward the request over to the remote server and give back the results to the Javascript code as if it originated at the same server. This is explained in detail in this Yahoo Developer Howto article. Being averse to adding more code than is absolutely necessary, I tried enabling mod_proxy and then mod_rewrite on my local Lighttpd webserver, but was not successful, so I ended up using a custom PHP proxy adapted from the code in the Yahoo article. This is shown below.

<?php
# searchmash-proxy.php
// Adapted from:
// PHP Proxy example for Yahoo! Web services. 
// Responds to both HTTP GET and POST requests (only GET for this one).
// Author: Jason Levitt
// December 7th, 2005
//

$url = 'http://www.searchmash.com/results/%query%+site:sujitpal.blogspot.com';

// Get the REST GET call from the AJAX application
$qt = $_GET['qt'];
$url = str_replace("%query%", $qt, $url);

// Open the Curl session
$session = curl_init($url);

// Don't return HTTP headers. Do return the contents of the call
curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);

// Make the call
$results = curl_exec($session);

// The web service returns JSON. Set the Content-Type appropriately
header("Content-Type: application/json");

echo $results;
curl_close($session);

?>

This proxy is called from the Javascript code. The search term is plugged into the URL, and the proxy builds the URL for the call to Searchmash, executes the request, resets the Content-Type of the request to "application/json" and spits out the text. To the Javascript code, it is as if this all happened when it called the PHP proxy. We did not need to change the Content-Type, but if we do, we can use Prototype's built-in text to JSON parsing functionality, otherwise we will have to eval(transport.responseText) ourself. The HTML page with embedded Javascript is shown below:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <title>My Blog Search Widget</title>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
    <script type="text/javascript" 
      src="http://prototypejs.org/assets/2008/1/25/prototype-1.6.0.2.js"></script>
    <script type="text/javascript">
function BlogSearch() {
  var request = new Ajax.Request(
    "/searchmash-proxy.php",
    {
      method: 'get', 
      parameters: { 
        qt : $F('q'),
      }, 
      asynchronous: false,
      onLoading: function(transport) {
        var html = '<b><blink>Searching...Please wait</blink></b>';
        document.getElementById('results').innerHTML = html;
      },
      onSuccess: function(transport) {
        var json = transport.responseJSON;
        var estimatedCount = json.estimatedCount;
        var term = json.query.terms;
        var results = json.results;
        var html = '<b>Total hits: ' +
            json.estimatedCount +
            ' for term: </b>' + 
            json.query.terms + 
            '<br/><br/>';
        results.each(function(result) {
          html += '<b><a href="' + 
            result.url + 
            '">' +
            result.title + 
            '</b></a><br/>' +
            result.snippet +
            '<br/><b>' +
            result.displayUrl + 
            '&nbsp;' +
            '<a href="' +
            result.cachedUrl + 
            '">Cached</a></b><br/><br/>';
        });
        document.getElementById('results').innerHTML = html;
      }
    }
  );
}
    </script>
  </head>
  <body>
    <p>
    <b>Enter your query:</b>
    <input type="text" id="q" name="q"/>
    <input type="button" name="Search" value="Search!" 
      onclick="BlogSearch()"/>
    </p>
    <hr/>
    <b>Results</b><br/>
    <div id="results"></div>
  </body>
</html>

The "Search!" button has an onclick handler that calls the BlogSearch Javascript function. This will make the call to the proxy with the content of the text input element. While the proxy is returning results, the anonymous function associated with the onLoading event will be called (simply setting the results div element's innerHTML to a blinking message, and once the response is available, the anonymous function associated with the onSuccess event will be called. Inside the onSuccess method, each result is parsed by yet another anonymous function, wrapped in a Ruby-like results.each() iterator. Finally the composed HTML is set into the innerHTML property of the results div block.

I copy both these files to the document root of my Lighttpd server and navigate to the HTML file (http://localhost:81/search-blog.html) on my browser, then enter the term in the search box and hit the 'Search!' button. Search results for the term 'json' are shown below:

There are several things I liked about this approach. First, no more futzing with browser detection and using XmlHttpRequest or its Microsoft cousin XMLHTTP directly. Second, the use of nested anonymous functions that improves the readability of the code. And third, the use of nested JSON objects to pass arguments to the function.

However, I felt the documentation for Prototype was rather sketchy. It is possible that I feel this because my Javascript is rusty, but this is likely to be the case for any newbie. Its not that the documentation is bad, its actually very well structured (much like Javadocs), it is just aimed at experienced Javascript developers. It may be helpful to have more examples of actual usage in the docs, much like the PHP docs on the net.