Getting familiar with Backbone.js

As a UX developer at Safari Books Online, I need to be familiar with all aspects of our front-end architecture. A big part of that is Backbone.js. My JavaScript experience is jQuery plugins and scripts so I’ve needed to play catchup on this Backbone thing. I learn best by pulling up my sleeves and diving into a project. I decided I’d like to work with some kind of API and ended up choosing the Google Books API.

Continue reading

Node.js is wonderful for robots

I was having a problem.

We use JIRA for ticket-tracking. It assigns each ticket a project code and ticket number, or key. I had been encouraging the team to refer to specific tickets in our chat room when discussing what they were working on, versus, “Hoping to finish up the thing with the error on the page.” My problem was that they started actually doing this, but there were now so many people collaborating that we had no idea what any individual “issue key” meant.

Them: Hey, Keith. I was noticing that there
  are a few dependencies linked to ZOMG-1337. 
  How's it going on BLOOP-9234?

Me: Uh...

We needed someone to help us with the menial task of looking up the associated JIRA ticket and summarizing the current status in all of our chatrooms. This sounded like a job for a robot. What surprised me was that it was also a perfect job for something other than Python, specifically Node.js.

An automated notification from GitHub triggers a response from jirabot

Continue reading

Documenting Multiple Programming Language APIs with Sphinx

The Goal

I wrote before about using Sphinx to generate documentation in multiple human languages, but Sphinx can also generate documentation for APIs implemented in multiple programming languages. Most programming languages have some kind of API documentation generation tool (either bundled with the language implementation or provided as a separate utility) for documenting software written in that language: Javadoc, JSDoc, RDoc, Doxygen, etc. These are really useful when you’re primarily working in a single programming language, but most of them start to show some limits in a multiple-language software project. If you’re writing a Django web application (Python) with a rich client UI (JavaScript) that leverages existing web services code (Java), then resources and flow of execution can be shared between languages. In this case, it’s nice to have a single searchable source of technical documentation, and with a little configuration Sphinx can do this.

Python

Sphinx was originally written as a tool for writing the documentation of Python itself, so it stands to reason that it has very good support for generating Python API documentation (in fact, it’s pretty much Python’s official tool for this purpose). The documentation for Python and Django are examples that many other projects follow, encouraging a style of documentation which reads more like a technical book than a raw listing of class and function descriptions. Here’s an example (from Django) of the type of wiki markup Sphinx uses for this:

Available ``Meta`` options
==========================

.. currentmodule:: django.db.models

``abstract``
------------

.. attribute:: Options.abstract

    If ``abstract = True``, this model will be an
    :ref:`abstract base class `.

Sphinx also includes utilities for auto-generating partial (autodoc) or complete (sphinx-apidoc) documentation for a Python API, fetching descriptions of each item from docstrings in the source code when available. For example, to generate reST files which describe all the Python code under a particular directory:

sphinx-apidoc -f -o docs/python src

This doesn’t take much work, and generates something that looks a little more like Javadoc output. Speaking of which…

Java

Javadoc was one of the first tools to really popularize in-source-code API documentation. There’s been some debate over whether this is really a good way to write the main API documentation for a software project, but at any rate, most decent Java projects include fairly complete documentation right in the source code. Using the Javadoc tool is by far the most common way to generate HTML documentation from these source code comments, but they can be used to generate Sphinx documentation as well. There’s a Sphinx extension called javasphinx which includes a tool to parse these comments and generate Sphinx reST files from them. Usage (after installing and configuring the extension as described in its own Sphinx-based documentation) is very similar to the equivalent Python utility described above:

javasphinx-apidoc -f -o docs/java src

And if you prefer the book-like style used for Sphinx documentation for projects like Python and Django, javasphinx provides a “domain” of reST markup extensions that can be used to describe a Java API in conjunction with wiki-formatted prose:

.. java:type:: public interface List extends Collection, Iterable

   An ordered collection (also known as a *sequence*)

   :param E: type of item stored by the list

(Example stolen shamelessly from the javasphinx documentation.)

The output may not be amazing enough to convince a Java development team to switch over from Javadoc output, but it has the big advantages of being combinable with other Sphinx source files (either hand-written or generated from other programming languages) and allowing generation of output formats other than HTML (such as PDF, EPUB, LaTeX, etc.) Additionally, the same techniques I described in my previous blog post for using gettext and Transifex to translate Sphinx documentation into multiple written languages can be used to translate the generated API documentation as well. The JavaScript search engine provided in the Sphinx HTML output is another nice bonus. Which leads us to…

JavaScript

JavaScript is a particularly important “second language” for a documentation tool because its virtual monopoly in web browsers means that developers of web services in many other programming languages also need to deal with it, but in some respects it’s one of the most difficult ones to support. Whereas most other programming languages have a standard enough structure that vaguely useful API documentation can be generated by automated tools even if the developer didn’t bother to comment his code, this has proven exceptionally difficult in JavaScript; unless a human identifies how the code is structured, software can’t reliably describe it in a way that’s particularly useful to humans. From this perspective, using the JavaScript domain that Sphinx provides for describing JavaScript APIs probably involves no more work than any of the alternatives:

.. js:function:: $.getJSON(href, callback[, errback])

   :param string href: An URI to the location of the resource.
   :param callback: Get's called with the object.
   :param errback:
       Get's called in case the request fails. And a lot of other
       text so we need multiple lines
   :throws SomeError: For whatever reason in that case.
   :returns: Something

(Again, this should look familiar to anybody who followed the link above.)

Nevertheless, many developers came to JavaScript from languages (especially Java) which support auto-generation of docs from source code, and heroic efforts have been made to enable this in JavaScript as well; a number of projects have endeavored to comment their source code in ways that work well with tools such as JSDoc, YUI Doc, JSDuck, and others. Of these, JSDoc is probably the most widely used so far, and there happens to be a fairly nice utility (by the somewhat long name of “JsDoc Toolkit RST-Template“) for getting JSDoc to generate reST files for Sphinx rather than outputting HTML directly. JSDoc uses the Rhino Java-based JavaScript interpreter, so the command for running this one is more typical of the Java world:

ant -Djs.src.dir=src -Djs.rst.dir=docs/javascript build

Some examples of its output can be found here. It unfortunately doesn’t seem to work with JSDoc 3 yet, adding support for that would be a nice project for somebody with a little free time.

Putting It All Together

The API documentation for each of the languages described above (and others as well) all get initially created as reST (reStructuredText) files. Once generated, they can be treated like any other Sphinx source file (although you probably wouldn’t want to edit them directly if you ever plan to recreate them from source). Phrases can be extracted for translation, they can be combined with each other and other documents to form a larger documentation package, they can be combined with the output of other cool Sphinx extensions (seriously, even this is only a partial list), and so on. You’d probably want to write some kind of automation script to handle the details for you (here at Safari Books Online I wrote a Django management command to do it), but once set up you have a very nice tool for generating and maintaining a pretty comprehensive set of technical documentation.

Taming JavaScript with Grunt

What I Needed It For

Some of my friends and I occasionally get together to play a pencil-and-paper role-playing game called Anima: Beyond Fantasy. This game has a really rich set of rules for creating the characters who inhabit the imaginary world in which the game’s story takes place. You can create just about any kind of warrior, martial artist, wizard, rogue, psychic, and so forth that you can think of from your favorite shows or books, complete with rules on how all of their abilities work in the game. But this flexibility comes at the cost of a pretty darn complicated character generation system; some abilities depend on having other ones first, there are limits on how much you can develop one ability in relation to others, and each character has a limited pool of “points” (of several different types) to be spent acquiring those abilities. Defining all the major characters in the story well enough to run them in the game turned out to be pretty time-consuming. Being a programmer, I figured this was nothing that a good piece of software couldn’t fix, and set about writing a character generation utility using a language (JavaScript) and a few libraries (jQuery, RequireJS, Twitter Bootstrap) that I wanted to get more experience with. About 18 months of occasional coding in my free time later, I had a mostly finished program…and quite a few JavaScript files. I’d kept the code fairly well organized, but now I wanted a nice automated process for generating an optimized version of the app that I could put online for other people to try out. Often for a project like this you’d use tools appropriate for whatever server-side web application stack you’re using, but this is a purely client application that runs completely in the browser as a single web page. I’d heard that people were using Grunt as a JavaScript build tool, so I decided to try that out.

Getting Started with Grunt

Grunt describes itself as a “task runner”; it’s basically a task definition and execution framework like make, ant, and rake —to name a few which are commonly used with other programming languages. It’s implemented in JavaScript itself, using the Node.js framework, so you need to install Node first in order to use it (this is becoming increasingly common for JavaScript-related utilities). Once that’s done, you use the Node.js package manager (npm) to install Grunt’s command line interface:

npm install -g grunt-cli

This basically just puts the “grunt” executable in your path, so you can run it in a project directory to execute Grunt tasks; it looks for the actual Grunt library inside whatever project directory you run it in. To be able to do that, we first need to make it look like a project directory. For Node.js to recognize it as such, we need a package.json file in the project’s root directory. For my character generation utility, it started out like this:

{
  "name": "anima-character-generator",
  "version": "0.5.0",
  "description": "A character generation utility for the Anima: Beyond Fantasy RPG",
  "homepage": "https://github.com/jmbowman/anima-character-generator",
  "keywords": [
    "game"
  ],
  "bugs": "https://github.com/jmbowman/anima-character-generator/issues",
  "repository": {
    "type": "git",
    "url": "https://github.com/jmbowman/anima-character-generator.git"
  },
  "devDependencies": {}
}

With that done, we can start installing Grunt and its plugins into the project directory. (These all get installed into a node_modules folder which you can add to your .gitignore file or whatever other mechanism your version control system uses to specify files and directories that shouldn’t be checked in.) For Grunt itself:

npm install grunt --save-dev

That last parameter tells npm to automatically add the library you just installed to the list of development dependencies for your project (basically, stuff you probably need when working on the code but not necessarily when just running it). So the “devDependencies” section of packages.json has now been updated to look like:

  "devDependencies": {
    "grunt": "~0.4.0"
  }

With that in place, other people who check out your code have a somewhat easier time getting things set up. They still need to install Node.js and grunt-cli as I described above, but to get Grunt and any other plugins for it you install later, they’ll just have to run “npm install” in the directory which has package.json in it.

Making Grunt Useful

Now that you have Grunt set up, you need to tell it what you want it to do. But Grunt itself really doesn’t know how to do all that much; it’s the huge selection of plugins for different kinds of tasks that really make this utility useful. So let’s install one of those first. I said I wanted to optimize my JavaScript files for deployment; I’m using RequireJS, so mainly I just need a plugin that’ll run that library’s r.js optimizer for me. Oh look, there is one:

npm install grunt-contrib-requirejs --save-dev

(Note that there are actually several Grunt plugins for RequireJS available. In cases like this where you need to pick one, the grunt-contrib-* plugins are usually a good default choice when available.)

The actual task definitions go into a Gruntfile.js (or Gruntfile.coffee, for CoffeeScript) file. To run my optimization task the way I want to, I need one that looks like this:

/*global module: false, require: false */
module.exports = function (grunt) {

    // Project configuration.
    grunt.initConfig({
        pkg: grunt.file.readJSON('package.json'),
        requirejs: {
            compile: {
                options: {
                    baseUrl: './js',
                    findNestedDependencies: true,
                    logLevel: 3,
                    mainConfigFile: './js/config.js',
                    name: 'libs/almond',
                    include: 'main',
                    optimize: 'uglify2',
                    optimizeCss: 'none',
                    out: './build/js/main.js',
                    wrap: true
                }
            }
        }
    });

    grunt.loadNpmTasks('grunt-contrib-requirejs');

    grunt.registerTask('default', ['requirejs']);
};

With this in place, I can type “grunt requirejs” (or just “grunt”, since I registered requirejs as the default task) and it will run r.js for me with the appropriate settings to concatenate all the JavaScript files ultimately used by my “main.js” script into a single file, replace the RequireJS loader with the lightweight almond loader (since I no longer need to load code from other files), minify the result with UglifyJS2, and finally create the generated file at the location specified by the “out” parameter. Note that I didn’t need to explicitly download r.js, write a shell or batch script to run it, create an “app.build.js” configuration file just to define the build, or a few other tedious things I otherwise would have needed to do. And the best part is that with this framework in place, defining new tasks becomes trivially easy and doesn’t involve adding any more new files to the project (we’ve only created two new files that actually get checked into version control). For example, if I want to generate HTML documentation for the JavaScript API from my JSDoc comments in the source code:

npm-install grunt-jsdoc --save-dev

And in Gruntfile.js:

        ...
        jsdoc: {
            all: {
                src: ['js/*.js'],
                dest: 'doc'
            }
        },
        requirejs: {
        ...

    grunt.loadNpmTasks('grunt-jsdoc');

Or to check all the JavaScript code (including Gruntfile.js itself) for common errors using JSHint:

npm-install grunt-contrib-jshint --save-dev
        ...
        jshint: {
            all: ['Gruntfile.js', 'js/*.js'],
            options: {
                bitwise: true,
                curly: true,
                eqeqeq: true,
                forin: true,
                immed: true,
                latedef: true,
                newcap: true,
                noarg: true,
                noempty: true,
                nonew: true,
                onevar: true,
                regexp: true,
                trailing: true,
                undef: true,
                unused: true,
                white: true
            }
        },
        ...

    grunt.loadNpmTasks('grunt-contrib-jshint');

These can then be run from the command line via “grunt jsdoc” or “grunt jshint”. The latter in particular I find extremely useful by making it trivially easy to check for and fix simple mistakes before even trying the code in the browser.

A Few Hours Later

In about 4-5 hours, I went from only having a vague understanding of what Grunt was to having a working Gruntfile.js that checked my code for errors, generated my documentation, concatenated and minifed my CSS and JavaScript files, renamed those generated files with MD5 hashes to avoid cache problems, and created a copy of the HTML page modified to work with those minified files (you can see the full file here). Granted, I already had scripts to run JSHint for a single file and to generate JSDoc output, and was pretty familiar with how r.js works. But using Grunt really did simplify the configuration and made it easier to run everything.

Oh, and if you’re curious about what that character generator looks like in action, you can find it online here.

Go as an alternative to Node.js for Very Fast Servers

Node.js has gained acceptance in mainstream software development at an amazing pace. There are a lot of good reasons for this: everyone loves software that is Very Fast, npm is truly an excellent package management tool/ecosystem, and its release coincided with the web development community as a whole awakening to the fact that JavaScript is a Real Programming Language. However, I think there are assumptions lurking in the subtext of the conversation around Node that are false, yet have contributed greatly to the excitement around it. I’ll whine about this briefly below:

  • Just because JavaScript is a real programming language doesn’t mean it is awesome. Crockford infamously has said that JavaScript is the only programming language that programmers don’t think they have to learn in order to use. I think that statement is less true today than even two years ago; both exposure to more serious JavaScript development and to a wider range of programming languages have taught us that we can view many JavaScript “bugs” — like function scoping, prototypal inheritance, and mutable execution context — as “features” that we can leverage to build better software. However, it is still a language where the commutative property does not always apply, where variables default to globals unless specified as locals, and where integer values do not exist. The focus in JS language development now is “more features with backwards compatibility in the browser.” Before I run this language on my servers, I want some fundamental design decisions overturned.
  • Conventional wisdom (and indeed the rationale called out by Ryan in his early slides on Node) says that putting JavaScript on the server means opening doors for front-end engineers to start churning out back-end code. Look ma, callbacks! I find this objectionable. If your JavaScript developer is capable of understanding JavaScript as a language, and has learned the DOM API, plus maybe some fancy HTML5 storage APIs, she is absolutely capable of learning a modern server-side framework in a different language. There may be emotional or psychological barriers to doing so, but not intellectual, and besides, learning new languages tends to broaden your understanding of the ones you already know.
  • Of course, it may be that you look down on your front-end folks and don’t consider their work “real engineering”. If you are in fact right, why on earth are you willing to give them a database connection? This thinking places too much burden on the tools and not enough on the user.
  • The biggest promise that Node makes is the ability to handle many many concurrent requests. How it does so relies entirely on a community contract: every action must be non-blocking. All code must use callbacks for any I/O handling, and one stinker can make the whole thing fall apart. Try as I might, I just can’t view a best practice as a feature.

Erlang, Scala, and Clojure are a few other platforms that have been touted in recent years for their scalability, but my favorite of this new pack is Go. Go was developed at Google by a couple of UNIX legends to replace C++ as a language for writing servers. As such, it delivers comparable speed to Node on a single-core machine and a much lower memory footprint (when configured correctly, Go will take advantage of multiple cores). This is not, however, what appeals to me about Go. While JavaScript drags the scars of its hasty standardization around with it, Go was designed very thoughtfully from the beginning, and as a result I find that it’s a pleasure to write. Though I’ve used Node and write a lot of JavaScript, I have gone to Go for servers for personal projects for the past year. Here are a few of the reasons I like it so much.

Static types with less, uh, typing

Go is statically typed. I personally love static typing, all performance considerations aside — it reduces the number of runtime errors I encounter and then number of unit tests I have to write, and is self-documenting by nature. The Go team mercifully designed a compiler that is quite smart about inferring types, which results in far less ceremony than you might expect.

Another aspect of the Go type system that I love is Go’s notion of interfaces. Interfaces are implemented implicitly. There is no “implements” keyword. Any type that defines the methods listed in a given interface type implements that interface by definition. This means that you can define interfaces that are implemented by code in the standard library, or in some other third party code that you don’t want to fork. The result is duck-typing that is compiler-enforced rather than simply a convention.

Shipping with the right tools

The Go team has gone out of their way to ensure that installing the language itself is all you need to start being productive with the language. Sometimes this means providing tools that you would expect the language community to produce after a few years. Sometimes it’s a question of making a decision on a contentious issue, so you don’t have to. And sometimes it’s just a great idea.

I particularly like Go packaging. The go get command fetches packages from any website that complies with a simple meta tag protocol, plus major source control hosting sites. All I need to do to make it possible for you to install my code is put a valid go package (a folder full of .go files that have the same package declaration) up on Github.

The Go source comes with syntax highlighting and other niceties for a number of editors, including an emacs mode and vim plugins. The vim plugins provide access to the built-in documentation generator (go doc) and the automatic coding style compliance tool (go fmt). go fmt parses your source and rewrites it using the AST and official formatting rules. There’s no arguing about whether to use semicolons, or putting your commas at the front of the line — the language knows what it wants. It’s a built-in hipster suppression mechanism.

An upshot of having command line tools that manipulate an AST is go fix, which will rewrite your source to accommodate API changes in the standard library or builtins if you upgrade your language version. I’ve only done this one, but it worked. I cannot get over how cool this is.

First-class concurrency

Go has first-class function values, so you could, in theory, prevent blocking I/O with callbacks (thought admittedly there would be a lot more cruft). But Go does much better than this: it provides keywords, operators, and primitive types that deal specifically with concurrency. This breaks down into:

  • The go keyword, which invokes a function as a goroutine — that is, a concurrently executing bit of code that you aren’t waiting on;
  • Goroutines communicate via channels, which are typed queue structures that are safe to share. These are basically like Unix pipes;
  • The <- operator, which as an infix appends a value to a channel, and as a prefix retrieves a value from a channel.

One of the creators of Go gave an excellent talk about concurrency patterns in Go last year, and I highly recommend watching it.

Robust standard library

Despite being a young language, Go has a very complete standard library. It addresses many of the needs of people who are writing systems software (there are 17 subpackages of the crypto package) but also more modern concerns like HTML templating with proper sanitization, JSON, and a really great HTTP package. There are also some officially maintained repositories outside of the stdlib that deal with newer protocols like websockets and SPDY.

This post could use more than one obligatory example, but if it has to be one, it should cover all my points above. We are going to port the first Node.js program I ever saw (from the slidedeck Ryan was using three years ago) to Go. It’s a small server that sends down 1MB responses for every request, perfect for testing handling of concurrent connections. The original looks like this:

  http = require('http')
  Buffer = require('buffer').Buffer;
  n = 1024*1024;
  b = new Buffer(n);
  for (var i = 0; i < n; i++) b[i] = 100;

  http.createServer(function (req, res) {
    res.writeHead(200);
    res.end(b);
  }).listen(8000);

We can write this program in Go with a similar line count and equal simplicity and clarity. It looks like this:

package main

import "net/http"

func main() {
    bytes := make([]byte, 1024*1024)
    for i := 0; i < len(bytes); i++ {
        bytes[i] = 100
    }

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write(bytes)
    })
    http.ListenAndServe(":8000", nil)
}

In this small space, we see type inference and use of the standard library. What you don’t see (by design) is that every call to the handler function passed to http.HandleFunc is invoked as a goroutine, thus handling any concurrency needs for us.

If you are using Node.js on the server and speed is a major factor in that decision, I encourage you to install Go and dump the code above into some files. You can run the Go server with go run yourfile.go. Run ab -c 100 -n 10000 http://{your ip}:8000/ in another terminal window, and use the tool of your choice to monitor memory usage. It’s pretty fast!

Now, if you’re not chicken, do the same with the JavaScript.

Detecting Available Free Space in WebSQL

At Safari, we’re working on some web products that allow offline usage. For various reasons, using the Application Cache isn’t appropriate for our use case; we need a “real” database. The browser requests the data from the server and then inserts it into the relevant tables in WebSQL. (Yes, I know WebSQL is deprecated, but unfortunately iOS doesn’t yet support IndexedDB and this functionality is particularly important on mobile phones, so that’s what we’re focusing on for the moment.)

The Problem

Generally speaking this works well, but you have to navigate the landscape of device capabilities, and there are times when it is particularly frustrating. Take for instance this scenario: I chose some content that has a lot of images, adding up to 18MB of stuff, and decide to save it offline. On iOS, we get 50MB of storage per domain (though the data is actually encoded as UTF-16, so it’s about half that in reality), so this should be OK.

So the content is requested from the server, streamed to the client, and the client attempts to save it in the database. What we didn’t know is that there’s already  12MB worth of stuff stored in the database. So what happens? After making the user wait to download the entire 18MB blob and then wait to parse it and try to load it all in the database (which could amount to a fairly large amount of time depending on a variety of conditions) – the database throws an error and we tell the user they’re out of space. I think you’ll agree that this experience is sub-optimal.

A solution?

Can we do better? I think we can. I want to preface this by saying that this most likely isn’t for everyone. It may very well be faster to just throw your data at the database and see if it sticks, but in our situation most of the time it will be faster to detect and then decide what to do. The problem is that there’s no built-in way to determine how much space is left before you’ve hit whatever cap there may be. Believe me, I googled quite a lot looking for a solution. After that search and chatting with some co-workers, we came up with something and that’s what I want to share: a proof-of-concept I’m calling yardstick.js. I’ve created a gist of the code on Github.

Using it

Yardstick works by using a temporary table to measure how much space is left by filling up to a given targetSize. Originally, I wanted to create a separate database and exploit the per domain storage limitationbut there were some issues with the allocation of the two different databases. Say we are in that situation above, we have already prompted for the max size (see here for why this is important), and we know that the content we want to insert is 18MB, usage of yardstick.js would look like this:

  var ys = new Yardstick();
  ys.measure(18, function(avail, err) {
      // continue what you wanted to do here
      alert(avail + 'MB available for saving');
    }
  );

What’s happening

An in-memory string of 1024 * 1024 characters is used to fill the database using targetSize for the number of loops. So we can assume that each loop inserts 1MB. On each successful iteration we set the internal available attribute to equal the result.insertId of the SQLResult returned, letting the auto-incrementing table do the counting for us. The result of this setup is that if we get to targetSize loops, we know that amount can be inserted, if we error before getting there, we have the internal available attribute that will tell us how much we can safely insert. Upon completion, it deletes the in-memory fake data and drops all of the records that were created, and then calls the passed-in callback with the value of available and the error object, if an error occurred.

Now we can determine if we can save that content locally before we make the (even more) expensive call to the server. And if we can’t save the 18MB, we know how much we can save so that we might be able to suggest saving alternate content that’s smaller.

What do you think?

This is currently just an idea and I’d love to get some feedback on it because I think there are others who know more about this than me. There are obvious downsides to this approach (speed being the obvious one; I also killed the browser on my iPhone 3GS trying to use it, and it probably has battery life implications) and I’m sure there are improvements to be made as well. In addition to the gist, I wired up a little playground. The first input will create a database and fill it with that much dummy data, you can then input a size you’d like to save and it will use yardstick.js to determine if that’s possible. If you have any feedback or want to talk about this, please leave a comment here, on the Github gist, or contact me on twitter at @meirish.

Working With WebSockets

Working With WebSockets

I’ve been tinkering around with Node.js quite a bit lately, trying to ramp up on a language that helps a front end engineer like me slice deeper into the back end and create more sophisticated applications. One of the side benefits of learning a new language is the gamut of novel technologies that a new language levies, such as WebSockets.

Node.js has a single-threaded, event-driven core. This translates into a language that allows for high concurrency, fast event-driven response, and easy round-trip client-to-server(-to-client) communication. Node.js is built in the same language that resides in the browser (JavaScript). This means that “real-time” applications that maintain state and communicate to the server without refreshing the browser can rely on the same browser-like listening behavior. In essence, the server acts very much like the browser, waiting for events to be called, and then operating when triggered on the callback.

This sort of call and callback functionality has traditionally been the realm of AJAX-driven applications. Pepper a browser page with some AJAX calls, which are routed on the server in whatever language, with response pushed back up to the browser usually in the form of JSON or XML, and parsed by the browser without ever refreshing the page. Asynchronous HTTP.

Enter WebSockets. If AJAX was the main driver behind the “Web 2.0” phase of the Internet (though definitions differ), maybe WebSockets could fairly be called version 3. Essentially, WebSockets are an HTML5 browser technology that define a full-duplex socket “handshake” over a single TCP connection. Translated, WebSockets provide a way for connecting from the client to the server with a constant shared connection. Think of it like the telephone. Dial the number, place the call, and once connected, voices can talk back and forth without interruption (until the call is ended).

Where does Node.js come into this? Honestly, any server-side language can employ WebSockets. (Scala, might do it via “actors,” for example.) What sets Node.js apart is that the event-driven behavior of the browser that WebSockets rely on is also native to Node.js, as both have APIs that rely on JavaScript. Put another way, the event handling API is not only native to Node.js (in Node terms, you can reference it as require(‘event’).EventEmitter;), but can be extended to handle back-end WebSocket functionality.

In fact, this is precisely what Node.js author-coder Guillermo Rauch did with his highly popular Node.js package, “Socket.io.” (For nerds/hipsters looking for a little more, “Engine.io” is it’s important twin.) Rauch took the Node.js events API and built a socket tool on top, complete with immediate event handlers and so forth. The whole nine yards.

A few of us at PubFactory recently tested Socket.io in a “Connect Four” engineering challenge. For sake of brevity, I’ll just state that implementation of a front-to-back-end connection was extremely simple to set up. In the matter of about an hour I was able to setup a very rough chat room that ran across the same IP address, over a fully functional WebSocket. (Unfortunately, Heroku, our deployment environment, does not support sockets, so we had to downgrade to XHR long-polling, which Socket.io easily allows, but WebSockets works locally in full splendor.)

Where will WebSockets take the web? My jest about “Web 3.0” may not actually be too hyperbolic. There’s already a growing list of developers and applications that utilize this stack. In fact, Node.js has become the nucleus for real-time frameworks reliant on WebSockets, including the likes of Meteor.js, Derby.js, SocketStream, which iterate far beyond Socket.io. These frameworks try and solve the problem, for example, of shared code (sockets and node combined tend to flatten the code stack), authentication over a constant connection, vertical and horizontal socket scalability, and other such problems. It’ll be interesting to see where it all goes.

Optimizing JavaScript in a Django Project: django-require

JavaScript code is becoming an increasingly large part of most web applications, but this often isn’t reflected in server-side web application frameworks. The core Django framework, for example, offers very little explicit support for JavaScript; it’s generally handled as just another type of “static file”, a catchall term for any file needed to render the site which isn’t Python source code or an HTML template.

The problem with this categorization is that modern JavaScript applications look more like traditional compiled software than static images; there’s source code, “compiled” (minified) code, compiler settings, dependency management, and frequent changes in the source that require rebuilding the compiled version. The Django staticfiles framework makes it possible to do all this in a separate deployment process, but that isn’t really ideal. For example, you don’t want to run all of your tests using the JavaScript source files, then discover after deployment that a bug in your minification process broke the compiled JavaScript files in production. The recently released django-require offers one solution to this problem.

Get organized

The first step in getting a handle on this is to organize the JavaScript code itself. The language doesn’t exactly lend itself to this very well, but through herculean effort a few decent solutions have been developed. One of the most popular is the RequireJS library, an Asynchronous Module Definition (AMD) loader. Instead of requiring you to list your scripts in exactly the correct order (sometimes difficult in a template inheritance hierarchy using a large set of JavaScript files) and hoping that none of them use the same global variable in conflicting ways (definitely difficult in a language where it’s actually hard not to accidentally define new global variables), AMD allows you to explicitly list the dependencies for a module of JavaScript code. For example:

define(['first', 'second', 'third'], function (first, second) {
    var exports;
    // My code which depends on first and second goes here
    return exports;
});

There are a few different ways of writing an AMD module, but this is one of the most common. Basically it lets you use standard JavaScript syntax to define a new module of JavaScript code, which explicitly depends on other such modules (but may not directly need their return values). This example depends on three other modules, but only directly uses code from two of them; the third was needed for some side effect (perhaps on the other modules or on the page DOM). The return value of the module’s function can then in turn be passed on to other modules which depend on it.

The big wins here are that your code’s dependencies are explicitly stated, you don’t need to add anything to the global namespace beyond the define() and require() functions provided by RequireJS, and RequireJS takes care of loading the dependencies when they’re first needed (and not executing the code in the same module twice). Of course, this is part of the core functionality of most other programming languages…

Dependency management

Anyway, you can now build up even a fairly large JavaScript codebase with confidence that you can keep each file to a manageable size and keep all the dependencies straight. But you wouldn’t want to put code like this directly into production; loading potentially dozens of little unoptimized JavaScript files to render a single page is hardly ideal.

To address this problem, RequireJS provides an optimizer called r.js. It analyzes all the dependencies of the main module, packs them into a single file, and uses a minifier on the result. You can even configure it to build multiple different top-level modules (for example, if each page has different scripts) and to break some of the dependencies out into one or more separate files (which can be useful if your scripts on different pages have some big library dependencies in common). If you do collapse all of the scripts on a page into a single file, you can use almond to shrink it even further by using a simpler AMD loader which doesn’t need to be able to dynamically load modules from other files. This lets us compile a version of our JavaScript that we’d be willing to serve in production…but how do we let developers test the compilation process locally to make sure deployments aren’t going to fail?

Enter Django

This is (finally) where django-require comes in. It’s a Django app which provides a mixin for staticfiles storage backends (as well as a couple of such storage classes already configured to use it, for convenience). When Django’s collectstatic management command is run to collect all the static files for deployment, this mixin adds a step which will run r.js in order to generate the optimized version of your JavaScript code. Because it’s integrated with the storage backend, the newly created optimized JS files are handled the same way as the rest of your static files (given a cache-breaking hash suffix when using require.storage.OptimizedCachedStaticFilesStorage, uploaded to Amazon S3 if using a subclass of storages.backends.s3boto.S3BotoStorage, etc.) And if you have Selenium or other browser-based tests utilizing LiveServerTestCase, you can just run collectstatic first and your automated tests will use the optimized assets which were generated, allowing you to test the minification process.

The main configuration settings for django-require are managed like any other Django settings; for example, your settings.py file might contain:

STATICFILES_STORAGE = 'require.storage.OptimizedCachedStaticFilesStorage'
REQUIRE_BASE_URL = 'js'
REQUIRE_BUILD_PROFILE = 'app.build.js'
REQUIRE_JS = os.path.join('src', 'require.js')
REQUIRE_ENVIRONMENT = 'node'

For convenience, django-require includes the r.js optimizer itself, almond, the Rhino JavaScript interpreter for Java, a simple default build configuration, and the require.js script (you’ll probably want to copy the latter into your project’s source code as a static file, possibly using the provided require_init management command). Your configuration for r.js (top-level modules, which minifier to use, etc.) goes into an app.build.js file (as normal for that tool), the location of which is specified using REQUIRE_BUILD_PROFILE as shown above. It also provides a template tag for including your main page script in a template in a way which works both in development with REQUIRE_DEBUG = True and uses the optimized files when REQUIRE_DEBUG = False.

If you have a more complicated setup involving different main scripts for each page (possibly in different directories) with a common script of base dependencies, setting the RequireJS configuration in a single place and having it work across all those scripts in both source and optimized modes can sometimes be a little tricky; I normally put the configuration in its own file (config.js), something like:

require.config({
  baseUrl: "/static/js",
  paths: {
    jquery: "src/jquery-1.7.1",
    underscore: "src/underscore",
    backbone: "src/backbone",
    text: "src/text"
  },
  shim: {
    "src/chosen.jquery.min": {
      deps: ["jquery"],
      exports: "jQuery.fn.chosen"
    },
    "src/jquery.cookie": {
      deps: ["jquery"],
      exports: "jQuery.cookie"
    },
    "src/jquery.placeholder": {
      deps: ["jquery"],
      exports: "jQuery.fn.placeholder"
    }
  }
});

This can be specified as the configuration for the optimizer easily enough in app.build.js:

mainConfigFile: "./config.js",

And then your base template can look something like this:

<script src="{% static 'js/src/require.js' %}"></script>
{% if REQUIRE_DEBUG %}<script src="{% static 'js/config.js' %}"></script>{% endif %}
<script>require(["{% script 'js/pages/base.js' %}"], function () {
    {% block page_script %}{% endblock %}
});</script>

A page which has its own script which depends on base.js would have something like the following in its template:

{% block page_script %}require(["{% static 'js/pages/page1.js' %}"]);{% endblock %}

This ensures that the base script finishes loading before the page-specific ones which depend on it, and that the RequireJS configuration is run first in development mode, before the location of the first script encountered might confuse the issue of what the baseUrl property is to be set to.

Once you have everything configured, you can test the site using the optimized JS files by running the following commands (remember to set DEBUG = False):

./manage.py collectstatic
./manage.py runserver --insecure

Now you can start writing those automated tests I mentioned earlier, to make sure things stay working. For example, you could write some Selenium tests for your Django site, as I described in some earlier posts.

Back to the Future

As part of the PubFactory squad coming over to Safari Books Online, I had some idea of the code that SBO engineers wrote but I really wanted to see “how they were living.”  What was it going to be like working in a Python/Django environment?  I set off to figure it out.  I cloned a pre-existing Safari Books Online repository and tried to run a project.  Concepts were very familiar, which was good, but I was surprised by something: there was not a trace of IDE configuration in the repo at all.  No hidden folders with project configuration, not even a mention of *.sublime-workspace in .gitignore.  The PubFactory team primarily uses Java so the IDE (mostly Eclipse and a little IntelliJ) is a big deal.

Slightly confused, I asked Liza what IDE her engineers use and she literally laughed at me: “IDE?? You mean editor?”  I didn’t mean editor, I meant IDE!  I had heard of a resurgence of engineers using lightweight editors from the 70′s (emacs, vi/vim) and apparently I now worked with a bunch of those people.  (Admittedly the word “hipster” did come to mind.)

I held this worldly bias until I had a Node app to build (more details in the upcoming days).  When I sat down to write some code I said to myself, “You don’t really want to write Node in Eclipse do you, who does that?”  I had Sublime Text 2 installed already so I gave it a go.  Turns out writing server-side code in something that is actually fast was really nice.  The soul-sucking 30 second Eclipse startup window just wasn’t there.

Soul-sucking screen

Soul-sucking screen

As a well-conditioned Java engineer I also started out relying heavily on Sublime’s code completion.  I was initially cross when it wasn’t great but there are Sublime plugins to make it better and it doesn’t take long to realize you don’t really need it either.  It was actually really nice to actually write the code as opposed to typing three letters, hitting ctrl+space and choosing your nugget of code from a list of options.

I went to a meetup a week after starting my Node app and heard about WebStorm — another hipster-friendly editor.  It was publicized at the meetup as having a “really nice debugger.”  In Java a working debugger is an absolute necessity, so I went to give WebStorm a shot.  My first impressions were that the editor was interesting (Live Edit feature is neat) and as publicized the debugger is nice. But then I thought, “Do you actually want to pick an editor based on its debugger?  You haven’t cared or needed it in the last ten hours.”  Another IDE feature melting away.

The natural conclusion to this story is I have now realized the error of my ways and dumped everything for the 70′s technology (MacVim).  I don’t think so.  MacVim is missing some things, or I haven’t figured them out yet. I still miss the ability to search through all code in your project without using grep.  I also haven’t quite gotten used to knowing where my code is yet (Eclipse makes that irrelevant with find a resource: cmd+shift+r).

I am currently in a halfway-house with Aptana Studio.  It’s an Eclipse variant — I know — that feels much faster than the full Eclipse install, especially Juno.  Aptana Studio has a built-in terminal so you don’t have to switch windows to see output, has useful git support, and lets me do all the Eclipse-y things I still care about.  Apparently this process will take a little bit longer with me.

 

[Ed. To be fair, some of our developers do use Eclipse, even for Python, but not the majority. — Liza]

Choose Your Libraries Wisely

At Safari we use a variety of JavaScript libraries. As we have expanded and started new projects the question: “how do we choose when to use what library?” has come up a few times. I’ll go through my general approach to choosing a JavaScript library and then talk over specifics around the three main libraries we use at Safari: jQuery, Underscore, and Backbone.

Deciding Which Library You Need

Generally you start with a problem. I first want to make an observation about jQuery plugins becuase I think it’s where a lot of people start when looking for a solution (I know it was for me). Many who already use jQuery will just get out their Google hammer and search for “jQuery + [insert problem here]“. There is a jQuery plugin for pretty much anything you’d want, but I think coupling all of your libs could lead to problems later. If all the plugin does is attach a few methods to jQuery.fn, then my advice would be to look for a jQuery-free version. The added benefit of a jQuery-free library is that if you ever decide to switch from jQuery to something else, you’ll have less code to convert or find a replacement for.

OK, so sometimes a jQuery plugin will be your solution, but I want to start earlier in the decision process than that: start by thinking about your problem and ask yourself “is JavaScript the right tool to solve this problem?”. If your answer is “yes”, ask yourself the question again. I’m not trying to be a jerk, it’s just that CSS3 can do a lot of things that used to solely be in the realm of JavaScript. Now it’s better to write a CSS class to do that and then use some lightweight JavaScript to add or remove that class when appropriate. Just because modern browsers can handle most of the JavaScript we can throw at them doesn’t mean our users (and their networks) should have to.

If JavaScript definitely is the answer, then there are a few criteria to use when deciding on a library.

Does it do what you want?

If it doesn’t do what you want, then obviously, you want to keep looking. The other thing to look at here as well is compatibility: will it work in the environments (mobile browsers, desktop browsers, node) you need it to?

Do you like the API?

Read through the documentation (no docs? we’ll hit that one next) and try and get a feel for how the library is used. Do you like the choices that were made during the API design? If you were to using this library on a daily basis, would you enjoy it?

Is there documentation?

While most of us are capable of digging around in the source to figure out what’s going on, it is great when we don’t have to. Just a list of API methods with their corresponding docstrings is often passable, but I prefer documentation that is more user-friendly.

Are there tests?

Having tests is a good sign in a project. It helps to keep the codebase stable and allow others to commit to the project more easily.

Has there been recent activity on the project?

Though this is not always an indication of quality, I’m usually more comfortable using a project that is actively maintained. It makes me feel like bugs in the project will get worked out in a timely fashion.

Deciding When to Use a Library

The three main libraries we use heavily at Safari are jQuery, Underscore.js, and Backbone.js. I think all of them meet the criteria outlined above, but knowing when to use each of them is not always straightforward.

jQuery

For me, jQuery is the tool I reach for whenever I have to do anything that involves

  1. DOM manipulation / traversing
  2. Event management
  3. Ajax anything

Lately I’ve also found jQuery.Deferred to be useful in some cases (and really fun to use). jQuery’s great because of it’s ease of use (I remember my initial reaction to jQuery being something like: “All I need to do to get started is read some documentation and throw around some CSS selectors?!”), and the cross-browser differences it smooths over.

More recently, I’ve been tinkering with the idea of only loading jQuery for older browsers that need compatibility and using something smaller like Zepto or Ender for my DOM needs. Though if this commit or this tweet are any indication, maybe jQuery’s 2.0 will be that smaller alternative.

Underscore.js

Underscore.js provides a bunch of utility methods for dealing with Arrays, Objects, and Functions in a functional programming style. The cases where to use Underscore are:

  1. When you want to use Array extras (Array.foreach, Array.map, etc.) in older browsers.
  2. Function.bind – Underscore’s bind and bindAll are handy in browsers where Function.bind isn’t an option
  3. When you need a lightweight template function. Underscore’s template is a slightly modified version of John Resig’s micro-template function
  4. Many other instances—really it’s just very useful, go take a quick skim through the documentation, there’s a lot there.

I like that Underscore is already pretty small (4kb minified and gzipped according to the site), so I don’t often fret too much about size when I think it might be nice for something I want to do. If you’re size- or performance-obsessed though, you might want to check out John-David Dalton’s lo-dash. You can do custom builds, and the focus is more about cross-browser consistency and performance where Underscore’s is generally more focused on spec compliance.

Backbone.js

Backbone relies on both jQuery and Underscore (or one of the alternatives I mentioned for each of those ie., jQuery can be swapped out for Zepto or Ender and Underscore can be swapped out for Lo-dash), so while it’s fairly small itself (6.3kb), there is a bit of overhead associated with its dependencies.

When do you use Backbone?

  1. Single page applications. These have become popular as of late, if you’re not sure what that phrase means, check out the examples on Backbone’s site – most of them are “single page applications” where Backbone is controlling the routing.
  2. When your code feels like a tangled mess. The first time I used Backbone was because I was looking for a way to organize a bunch of spaghetti jQuery code. Backbone’s abstractions of View, Model, Collection, and Router often fit with things you’re already doing once you get to a certain complexity in a JavaScript application.
  3. If you’re trying to manage a lot of events, Backbone’s View is very handy.
  4. If you need to control and respond to browser history with hash URLs or using pushState, Backbone’s Router is worth looking at.

Backbone is a MV* framework, one of many in JavaScript, but what I like about it is its flexibility and the fact that’s it’s fairly bare-bones (seing a common pattern here?). It’s just enough to do what you want and then get out of your way. Some think this leads to too much boilerplate code, but for me, it lets me do things the way I want.

We have other libraries we use as well, but some combination of these three form the base for most of our projects. They might not work for you and your team, but I highly recommend settling on some standard libraries so that don’t have to learn a new library on each new project.