Teaching Physical Computing with Arduino

groupshot

David, Theo, Me, Josh, Kazu, Roberta & Archer

Recently, my infatuation with Raspberry Pi, Arduino, and Physical Computing coincided nicely with Safari Books Online’s dedication to ongoing education and involvement with ConnectEd. Having volunteered at Seabury Hall in Makawao, Hawaii (Maui) in the past, I was given the opportunity to submit an idea for a “Winterim” 4-day workshop on the topic of my choice. If enough students signed up, I would spend a week in Maui teaching the workshop (and have a few days on the beach, of course).

Bill is busy turning his workshop into a free Arduino Bootcamp, which will be featured here on the Flow blog. If you’d like to be notified when it starts, follow us on Twitter, Facebook, or Google+.

Realizing that Raspberry Pi may be a little too finicky for middle schoolers, I shifted my attention to Arduino. I turned to Safari Flow to learn Arduino myself, and then used it to build a syllabus that matched my own educational journey and discovery curve. After submitting my proposal to the school, I anxiously waited to hear how many students I would get. The fact that alternative workshops were on topics such as sailing, canoeing, hiking, and windsurfing didn’t help my anxiety! I was pleased when I learned that 6 students had signed up. Almost immediately, I started receiving questions from the enthusiastic students, such as “Can I bring in my box of junk electronics to make a robot?” and “Can we hack my remote control car?” Once I saw those questions, I knew we were going to have a great week.

With the books and videos that I had in Flow and a few rather large orders from Maker Media and Adafruit, I was able to gather enough course material to ensure that we eased into programming and electronics while laying a strong foundation for tinkering, setting up the students for continued learning beyond the 4-day workshop.

An overview of my workshop is below, and in the near future I’m planning to share specific chapters and videos that were useful to me. I’m hoping that our story and pictures below will inspire you to use the material to support students of your own.

Day 1: Basic Circuits and Blinking LEDs

Kazu sets up

Kazu had the most complex breadboards, but somehow always knew exactly which wire went where, and what it did!

On the morning of the first day, we built some basic circuits without a microcontroller to get a feel for prototyping circuits using tactile buttons, potentiometers, force sensitive resistors, and ambient light sensors. In the afternoon we went over to the lab to demonstrate how a microcontroller can do many things with the same basic circuits, using code to modify blink patterns, blink durations, and multiple LEDs. After some more light coding, we were able to play a melody using a piezo-buzzer via pulse-width modulation.

Day 2: More Inputs and Outputs

Josh

Josh prepares to upload a sketch

On day two, we used a third-party library and switch statements to decode signals via an infrared receiver, which listened for commands from universal remotes to control various outputs. We were able to switch on LEDs, control servos, and change colors of an RGB LED. We learned to read and sketch schematics and dug into basic programming concepts like switch and if/else statements and for loops. We also got a lot of practice mashing up code samples from books with our own code and modifying them to achieve our desired functionality.

Day 3: Field Trip

Drones in the courtyard

Drones in the courtyard

Unfortunately, I don’t have pictures from the field trip (the exact location is “classified,” sort of), but we saw a web-controllable telescope, which was part of the DoD telescope array for tracking US and foreign satellites. We also got to check out the Boeing electrical systems integration room, where there were “real live” electrical engineers designing and soldering circuits to create custom, rack-mounted motor controllers for various telescopes. In the afternoon, we built “brush bots” from a toothbrush, vibration motor, coin-cell batteries, and LEDs. We then let the bots fight it out. We also took turns flying a Parrot AR drone!

Day 4: Show and Tell

Archer demonstrates arming and disarming his intruder alarm with a universal remote

Archer demonstrates arming and disarming his intruder alarm with a universal remote

On the final day, each student combined what he or she had learned to build a personal project. Here are some examples of what they came up with. It was amazing to see what they were able to design in a 5-hour day after only 2.5 days of instruction:
  • A temperature probe that turns on an LED when the temperature exceeds 23° Celsius
  • A game that challenges you to press a button as many times as you can (counted on a 7-segment display) until your toast pops up, as detected by a PIR motion sensor
  • A fortune-telling “magic” 8-ball: Ask it a question and push a button on the remote, which triggers a servo pointer to randomly move back and forth before stopping at a response on the dial
  • A motion-activated intruder alarm that can be armed/disarmed via remote control
  • A motion-activated intruder alarm that plays a threatening melody with a buzzer when motion is detected

Special Thanks to Jaqueline Peterka and Roberta Hodara at Seabury Hall in Makawao, Hawaii, for making this happen!

Making 3d-printable objects for programmers

In the United States, we are just entering the holiday season, so I’m going to tell you a story about me being unbearably geeky. I’m doing this as a service so that you can interact with the geeks in your life and to share what I’ve learned about some technologies that let you design and print gifts and ornaments. Unless the geekiness in this post makes perfect sense to you, in which case I’m doing this in order to let you know you can use code to create physical objects, which is interesting in its own right.

Continue reading

Learn Networking with a DIY home firewall

Ed. note: Even blogs get holidays off. Please enjoy this reprise post from October 31st, just before November’s Blog-a-thon kicked off. Happy Thanksgiving!

Background

Ever since I’ve had a home internet connection, I’ve used consumer grade routers, like linksys or netgear (using both their factory firmware as well as dd-wrt). A few years ago I upgrade to the Apple Time Capsule. This gave me a stable and reliable wifi router with wireless backups. My home network has grown to support multiple phones, tablets, laptops, desktops, and servers running VMs. With all these devices, I wanted the ability to VPN into my home network. Looking online and at consumer electronic stores, home VPN routers cost between $150-$500 and received awful reviews. Most were proprietary and clunky. Low-end VPN routers lacked good software and hardware. I spoke with Clark Hartsock about this issue and he suggested running my own pfsense box. I liked the DIY aspect of this approach, and I am always curious to explore a new technology.

Continue reading

My GPS disciplined Rubidium OCXO Frequency Standard and NTP server project

Part 1

The Doctor: People assume that time is a strict progression of cause to effect, but actually from a non-linear, non-subjective viewpoint—it’s more like a big ball of wibbly wobbly… timey-wimey… stuff.

Amy Pond: It’s spacey-wacey, isn’t it?

—Dr. Who?

Though delightfully silly, there is some truth to that quote! Time is an amazing phenomenon to me. Concepts of time—whether philosophical, subjective, or rooted in the equations of Modern Physics—exist at the core of so much that governs our lives. Time is a measurement like the orthogonal dimensions of length. Everything in the universe contains a dimension of time. But time is also a where as much as a when. The relative experience of time that any object is at once unique, yet, within some conditions, is still relatable to other spacey-timey stuff around it. We can sense time, but we can’t hold a “1 Second” of anything in our hand.

Continue reading

My Quest for Circuit-Board Immortality

My Quest for Circuit-Board Immortality

I don’t know about you, but after painstakingly building a simple circuit to control a Raspberry Pi soundboard for my kids’ train set (inspired by this Make Article) on a breadboard and debugging code to make it work how I wanted, I got a little too attached when it came time to break it down and re-use the breadboard for my next creation. I knew full-well I would never come back the the soundboard if I had to re-wire the whole thing every time.

A Raspberry Pi for each project was impractical, as was a drawer full of breadboards with loosely attached components that toddlers would love to mangle. So, I set out looking for a better way to immortalize my creation and make it more modular so I could easily switch back and forth between “completed” projects. Here were my options:

Etch my Own PCB

Etching PCBs at home looks like a really fun and worthwhile project. I want to give it a shot at some point, but I didn’t like the idea of the chemicals with kids in the house and I wasn’t too keen on drilling so many tiny holes without a small drill press. I also didn’t like the idea of trying to go double-sided without through-plated holes (more in this later). Next!

Stripboard or Dead Bug Style

Started, but decided perfboard was not for me

Started this route, but decided perfboard was not for me

Dead-Bug style or Stripboard seemed to be quick and dirty approaches: boards are cheap and available locally, but it just wasn’t working for me on this project. I had a hard time routing and trimming leads to the right length or getting components to stay in, and I was kind of turned off by having to smear big blobs of solder to connect my components. Plus, the final product just wasn’t “pretty” enough because, let’s face it, I wanted to show this thing off! Next!

Professionally Printed PCB

Introduction to Fritzing for designing custom PCBs

Introduction to Fritzing for designing custom PCBs

I am not new to CAD but I’m definitely new to designing circuits, so I was instantly attracted to Fritzing.org‘s easy-to-learn Mac App. Fritzing lets you drag and drop components from your virtual Parts Bin like you would grab from your real stash, lay out your traces and vias (holes which are through plated for making connections between sides of the PCB) in the top AND bottom views, and see your cost to print in real time as you resize the board. This adds an extra fun challenge of trying to keep the cost down by optimizing your surface area, top & bottom, while taking advantage of the through-plated holes to connect the layers where needed. The software has a wide range of standard components as well as components from leading parts suppliers, and also lets you lay out the IDs of each component or place your own custom silk screening. When you are confident your circuit is complete, just a few more clicks plus your a credit card number and the PCB will be created and shipped straight to your door.

Some tips

I had a hard time starting with Schematic or Breadboard layouts, as I couldn’t get my components to line up when switching to the PCB view. Instead, I just designed everything in PCB mode without looking at the others and took a gamble that my circuit was laid out properly because I just couldn’t keep them nicely in sync.

After a short — OK, a little long — 2 weeks, I got my boards and they were much cooler than I ever expected! Now I have a pretty, somewhat more kid friendly, plug and play controller that enables me to come back to this project with minimal hassle. Take note of the date of the next production run and get it in by the deadline to help speed up your delivery. Plus, a deadline is always motivating.

Front of my Board

Front of my Board

Back of my Board

Back of my Board

Jumper Wires hacked into 26-Pin Socket, fits right into GPIO

Jumper Wires + right-angle headers hacked into 26-Pin Socket, fits right into Pi’s GPIO

Front assembled - Not bad for my first soldering project in Many years

Front assembled – Not bad for my first soldering project in Many years

Lessons Learned:

If I had to do it over again, I would still design my own PCB and have it printed professionally, rather than an off-the-shelf solution. Next time, I would make a bigger effort on a more sturdy pi->pcb interface to use with this and all future projects to make them much more “pluggable”, like a homebrew ethernet adapter, similar to this Patch Shield, or some variant of a built-in 26 pin ribbon socket to the custom PCB like Adafruit’s very tempting Perma-Proto.

Also, I should have probably left room for mounting holes and since the silk-screening turned out so nice, I would spend some time making the board prettier.

Future Hardware Projects:

  • Mount PCB in a wood and plexiglass enclosure
  • 3D Printed Pi “Base Station” for my growing collection of custom controllers
  • External LCD to display a simple menu of all my programs that match my controllers for further modularity

Resources:

Hands-on Electronics, from Practical Electronics for Inventors

practicalelectronicsforinventors

Moving from Breadboard to Prototype, from Distributed Network Data

distributedNetworkData

Soldering On, from Arduino For Dummies

9781118446430

Making Things: Electronic, from Arduino Robot Bonanza

ArduinoRobotBonanza

Meet Chirpy – The Raspberry Pi Dancing Bird Subscription Notifier

What the heck is a Dancing Bird Subscription Notifier?

Like many companies, Safari’s projects are given an internal secret codename that follow a pattern. In Google’s case that pattern is alphabetically ordered confectionary; in our case it’s bird names.

With Safari Flow, an all-new product, we re-architected many of our systems, and we decided to build the user database in Salesforce.com. I realized that we could have some fun whenever a new subscriber joined, alerting the team in the San Francisco office to the good news. So we (OK, I) built Chirpy as a side project, unknown to the rest of the team. Chirpy is a dancing mechanical “bird” who alerts us in a spectacular manner for each new subscription to Safari Flow.

The short version is that whenever a new subscription is created in Salesforce, that event triggers an email to an internal gmail account, which is then picked up by Chirpy. Chirpy – built out of a combination of an old wine box, some oak flooring scraps, a Lego Gearbox, a Raspberry Pi, powered speakers and a lot of cardboard, wire and hot glue – then plays a mating call and spins its spirograph (early Flow Beta testers may recognize these?), all while flapping its wings and flashing some LEDs (Flow colors, of course).

Early versions of Safari Flow used spirographs instead of book covers

Early versions of Safari Flow used algorithmically generated spirographs instead of book covers

Here’s what happens when you make Chirpy happy:

For those of us in the office when it happens, this is truly motivational, so keep those subscriptions coming!

The longer version of how Chirpy came to be follows, along with links to the individual chapters and books I used to help me (from Safari Flow of course)

Continue reading

Nagios Plugin for Arduino Environmental Monitoring

Last November I wrote an article explaining how to capture temperature data with an Arduino. These data were sent to the serial bus on a computer via USB. Now that the data are on the computer, we can track and monitor their flow.

Firstly, we should store these data in a database. I rewrote the serial reader to dump data into mysql.

import serial
import time
import MySQLdb

dbhost = 'localhost'
dbname = 'DB_NAME'
dbuser = 'DB_USERNAME'
dbpass = 'DB_PASSWORD'

ser = serial.Serial('/dev/ttyACM0',9600,timeout=1) # On Ubuntu systems, /dev/ttyACM0 is the default path to the serial device on Arduinos, yours is likely different.
while 1:
    time.sleep(10)

    the_goods = ser.readline()
    str_parts = the_goods.split(' ')

    conn = MySQLdb.connect (host = dbhost,
                    user = dbuser,
                    passwd = dbpass,
                    db = dbname)
    cursor = conn.cursor ()
    sql = "INSERT arduino_temp (temperature) VALUES ('%s');" % (str_parts[0])
    try:
        cursor.execute(sql)
    except:
        pass
    cursor.close ()
    conn.commit()
    conn.close ()        

    print the_goods 

Next, I want to track this sensor feed. I have a nagios server so I wrote a plugin to check the arduino output in mysql.

#! /usr/bin/python

import datetime
import time
import MySQLdb
import sys

dbhost = 'localhost'
dbname = 'DB_NAME'
dbuser = 'DB_USER'
dbpass = 'DB_PASSWORD'

# Make sure this file is executable. You should chmod +x sensor_status.py

def main():
    
    mysql_datetime = ""
    current_temp = ""    

    conn = MySQLdb.connect (host = dbhost,
                user = dbuser,
                passwd = dbpass,
                db = dbname)
    cursor = conn.cursor ()
    sql = "select created_at, temperature from arduino_temp where created_at = (select max(created_at) from arduino_temp);"
    cursor.execute(sql)
    rows = cursor.fetchall()
    for row in rows:
        mysql_datetime = row[0]
        current_temp = row[1]
    cursor.close ()
    conn.commit()
    conn.close ()        
    
    timediff = datetime.datetime.now() - mysql_datetime
    
    if int(timediff.total_seconds()) < 15:
        print "Shed was %s Fahrenheit and checked only %s seconds ago" % (str(current_temp), str(timediff.total_seconds()))
        sys.exit(0)
    else:
        sys.exit(2)


if __name__ == "__main__":
    main()

Nagios is monitoring software that uses a really simple algorithm for assigning status. Nagios runs your bash/python/whatever script and looks for an exit code of 0-3. Zero means everything is all right, 1 is a warning, 2 is critical, and 3 is unknown.

Nagios has a simple configuration that defines command and services. The executable sensor_status.py script is defined with the following command.

define command{
	command_name	arduino_temp_sensor
	command_line	/YOUR_PATH_TO_FILE/sensor_status.py  # chmod +x sensor_status.py
	}

The machine connected to the arduino sensor runs a service that calls this command.

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       localhost
        service_description             Shed Temperature
        check_command                   arduino_temp_sensor
        }

Sensor_status.py checks if the current system time and the last mysql record are more that 15 seconds apart. The arduino serial_reader_mysql.py inserts data every 10 seconds into mysql. This fifteen seconds check is enough time to know if data are not flowing properly.

If everything is working, Nagios will look like this.

Nagios_screen_shot

As you can see, monitoring your arduino temperature sensor is relatively simple. Nagios is a great way to make it happen. You can get all the source code to this project at https://github.com/muskox/arduino_env_monitoring/

Touch Input and Responsive Web Design

With the release of Windows 8, touch has finally landed on the desktop. There have been some great posts about the design implications of this change, but I haven’t seen much about technical implications, so I decided to write this. Chances are you’ve thought about  touch interaction if you were building a separate mobile site, but what does it mean for a desktop site or a responsive site?

Why Optimize for Touch?

Why is this something you’d want to do? It boils down to making things better for those who visit your site regardless of the device they’re using. This is why many people and companies have chosen Responsive Web Design. But what does Windows 8 have to do with any of this? It’s the first desktop operating system to include touch (and pen input, but more about that later) as an OS-wide, first-class input mechanism. So suddenly, if you have touch hardware, your desktop browser is going to be firing real touch events instead of using a third-party software abstraction to fire mouse events in the OS.

Who should be checking to see if their site needs any changes? If you’re doing anything with JavaScript click-handlers – most sites these days are doing something - then there are some touch-related things to think about. Additionally, maybe you want to consider gestures of some sort or you want to incorporate two or three-finger touch events. But the most important thing to consider is how you handle the simple single-touch event. Often you’ll want to trigger a touch event instead of click.

I’m going to focus on this last one as I think it’s the most common case. The reason single-touch needs “fixing” is that click events incur a penalty in the form of a 300 millisecond delay on many touch-enabled devices. This is to allow the OS to determine if you meant to pan or zoom instead of just clicking. You can speed up this interaction by checking those things yourself.

Optimizing (allthethings) for Touch

There are a few ways to do this. You can feature-detect touch events and then bind to “touchend” instead of click. (Modernizr is great for this but may be overkill for this one use case.) If you decide to go this route you will have to manage everything yourself: handle touch, determine if the user is scrolling/panning/zooming (if they are: don’t fire the handlers, if they’re not: fire it), and then after you fire it there will still be a click event fired.

What—you were expecting this to be straightforward? We are dealing with web browsers after all! What can we do to make this easier? Use a library, of course! But before doing that, I recommend reading through this Google post (this was the first place I read about the technique) so that you understand everything involved with overriding the default browser behavior. The library I’d recommend using is FTLabs’s FastClick. Be sure to read through the source too, it’s well documented. The nice thing about FastClick is that you only have to bind it once for the whole document and it only inserts itself if your device supports touch, as it has the feature-detection built in.

The Touch Future (or: When Touch is Not Enough)

Touch and click events are closely related as there was some effort to have backwards compatibility. But leave it to IE to break up the party. Turns out IE10 and “Metro” apps don’t have touch events (read more about the problem in this Modernizr issue on Github). Instead, Microsoft has abstracted and unified input mechanisms with what they’re calling pointer events. Instead of binding to touch or click, you bind to pointer event and the OS knows when to fire it based on the input method you’re currently using (touch, mouse, pen, laser, etc.). Boris Smus has written a polyfill for pointer events and has a great post explaining it, though I think it might not be ready for primetime. Recently, the W3C has formed a committee to begin the standardization process for pointer events, so it’s definitely something to keep an eye on.

I think pointer events will be the way forward because they are a better abstraction, and you don’t have to do a bunch of hokey pokey to get things to work properly, you simply let the browser do the heavy lifting for you. Even if that’s true though, touch events aren’t going anywhere anytime soon. They’re something you’ll need to feature-detect (there’s talk of defining Modernizr.pointer or Modernizr.mspointer in the above Modernizr issue) and then do what web developers always do: treat IE differently.

Distributed ebook file conversions with Amazon EC2: Part 1, the stack

It’s Friday afternoon. Your new ebook product is launching on Monday and you have 1,000 files to automatically convert from a legacy format to EPUB. You figured it would take a minute or two per file, but that was before you tried the converter on some actual commercial ebooks, the kind with lots of really big images. It turns out that if you converted all the books one at a time on your laptop like you’d been planning, you’ll be ready to launch in 2015.

Don’t panic. There’s always EC2.

Amazon’s cloud-for-hire service is widely used by internet startups (and post-startups), but it’s a potential lifesaver for any kind of large-scale computing task. Like it or not, the publishing industry today is about producing computer-readable data. Even if you just make a couple of EPUBs a year, you’re developing software, and if you make several hundred EPUBs a year, you should probably be thinking about scalable data-crunching techniques.

In this post, I’ll describe the data conversion problem and the architecture I used to crank through almost a thousand files in a couple hours. In part 2, I’ll walk through the Python code that powers all the pieces and assess the cost/benefits of this kind of approach. Programming skills are assumed.

Entering the EC2 world for the first time is intimidating, even for an experienced developer. There’s a lot of terminology, and books and articles get out of date fast. This post isn’t a comprehensive tutorial, but I’ll try to point out the places where it’s easy to get stuck or confused.

For my ebook conversion project, I had to solve several problems:

  1. I had to process a lot of files in a short amount of time; that was impossible even with the fastest single computer I could find. So I’d need parallelization.
  2. I had to assume that my conversion process might have bugs to be fixed or improvements to be made in the future. So I needed repeatability.
  3. I didn’t intend to spend my weekend turning on and off hundreds of servers, and the penalty for leaving servers running unnecessarily could be a very expensive bill. So I needed server provisioning to be scriptable.
  4. Once I spun up my servers in parallel, I needed a way to distribute the work across them in an orderly way. I didn’t want to waste time and money converting books unnecessarily.

Why EC2?

  • It’s metered. You can rent very, very powerful hosts for only as much time as you need.
  • It’s scriptable. Once you get the basics set up the way you like, it doesn’t much matter whether you want to run the same process 1 time or 1,000,000 times. I turned to the widely-use boto Python library to script my EC2 provisioning process.
  • It’s repeatable. This is more than just about being scriptable; you can create a snapshot of the host exactly as you want, with all of the dependencies pre-installed. If you need to run the same process again in a year, nothing will have changed.

The one thing that EC2 couldn’t magically solve for me was the distribution problem: each EC2 virtual computer would be a newborn baby, with code all ready to go but no idea which files to process. After flailing around for awhile I settled on using a queuing service. Amazon includes Simple Queue Service with their offering so I was all set.

Customizing the EC2 instance

An Amazon Machine Image (AMI) is your virtual computer. AMIs are a complicated subject, but for general purpose Linux needs I pick one of the official Ubuntu AMIs. Customize the AMI storage/memory allocation to match your application’s needs. In my case, I needed a fair amount of both storage and RAM as the conversion process generates a lot of temporary files, but it turned out I didn’t need the most robust processor. (If you need a lot of storage or want lots of files pre-installed, you’re going to also need to configure your own elastic block instance and mount it; avoid this step if you can because it’s a pain.)

User data

You can (and should) install all the dependencies you need on your AMI before taking a snapshot. However, if anything about your application changes, you’ll need to rebuild the AMI, an annoying and non-automatic process. User data, a feature available on some instance types, is a simple user-authored script that will run on boot, and can be provided on instantiation. Since my converter’s source code was available on a private Github repository, I set up my user data script to automatically pull the latest version of the code, which was pre-installed with the correct SSH keys in the AMI:

#!/bin/bash

cd /home/ubuntu/ebook_converter

# User data scripts run as root, so ensure that we "su" to the local user account
sudo -u ubuntu git pull origin master

My conversion script runs as a Python program inside a Python virtual environment. It’s possible that the latest version of the code has new dependencies, so after the instance pulls the latest code, the user-data script also runs setup.py develop:

su ubuntu -c "source ve/bin/activate && python setup.py develop"

The job queue

To solve the workload distribution problem, I needed a centralized place to store all of the books to be converted. Originally I thought I’d spin up each EC2 instance with a predefined set of books to convert and pass that list in to the user-data script, but I realized a far better approach was to use a queue. A queue is just a data structure that maintains an ordered list and allows items to be added or removed programmatically. If I’d provisioned the work list for each instance myself, I’d have to deal with potential problems like an instance crashing during processing and leaving half its workload untouched. Better to leave that to the queue.

(The AMIs and the queue need to be in the same Amazon region or they won’t be able to find each other.)

The final pipeline

I ended up with a workflow like this:

  1. Get a list of books to convert.
  2. Push the list of filenames to the queue.
  3. Start up the maximum number of simultaneous instances (20).
  4. For each instance:
    1. Update itself with the latest code.
    2. Get a filename off the queue.
    3. Go get the file.
    4. Convert it.
    5. Put it somewhere.
    6. Repeat.

Each instance keeps requesting books off the queue until the queue is empty; at that point it can just shut itself down. If an instance crashes, there are still others to pick up the slack.

The entire workflow can be written straight into the user-data startup script. I ended up with this script, which self-updates the code, runs “forever” (until the queue is exhausted), and then turns itself off:

#!/bin/bash

cd /home/ubuntu/ebook_converter

# User data scripts run as root, so ensure that we "su" to the local user account
sudo -u ubuntu git pull origin master

# Invoke the Python virtual environment and update any Python dependencies
su ubuntu -c "source ve/bin/activate && python setup.py develop"

# Invoke the virtualenv and run the process
su ubuntu -c "source ve/bin/activate && python ebook_converter/process_from_queue.py"

# Once this finishes, we're out of jobs on the queue, so shut down the host
shutdown now

EC2 lets you remotely monitor any server, so if your job writes logging information, you can watch that in real time. Here’s the output from my converter as it starts up and picks a file off the queue to download and convert:

Or better yet, take advantage of a program like multitail and watch all the log output simultaneously! Shortly after starting up 20 instances, my terminal window looked like this:

The primary purpose of running multitail is to feel like a total rockstar and be able to email this screenshot to all your coworkers. Or better yet, send a photo of you drinking a beer in a hammock while 20 computers in the cloud slave away on your behalf at the click of a button.

In the next post I’ll walk through the Python code that powered the provisioning, distribution, and queuing of the jobs, as well as the economics and practicality of this approach.

The little Pi that could

‘Big Iron’ computing is fun. SANs, fiber, clusters and data centers are where my work days are spent – and I love the work!

At the same time, ever since I wire-wrapped my first z-80 based computer, I have been captivated by the potential of small, affordable, and modestly powered computing. In the past, single board computers based on mainstream (x86) technologies have been interesting, but were usually significantly more expensive than a hobbyist/tech tinkerer/product inventor (dreamer!) could afford.

More recently there has been a surge of interest and creation of a wide range of embedded or single board computing options – from both the commercial space and Open Source communities. Examples of these include the Parallax Basic Stamp, Beagle Board, Gumstix, and many more.

The two platforms that have most recently caught my imagination are the Arduino and the Raspberry PI. The Arduino has been extraordinarily successful at providing an affordable, flexible platform for embedded controllers and dedicated single function computational platforms. What has been missing, until recently, is the general computing equivalent of the Arduino.

When I first read about the Raspberry Pi – that in addition to their original goal of an affordable education platform, “Developing countries are interested in the Raspberry Pi as productivity devices in areas that simply can’t afford the power and hardware needed to run a traditional desktop PC; hospitals and museums have contacted us…” – I thought to myself, “That’s cool, but I think they are optimistic about the $35 price point.” Then, of course, I ordered one right away.

While waiting for its arrival I considered a number of options to put this little board to the test. Then while following the Twitter streams for a number of co-workers attending the Books In Browsers conference in San Francisco this year it hit me – I must get our Ibis Reader EPUB 3 browser-based book reader working on my Pi!  I see this as an interesting blend of browser reading with an added potential for open access to information and the opportunities created by lower-cost computing.

Doing the installation seemed reasonably straightforward. There are several OSs available for the Pi and the ARM based Debian distribution called Raspbian is very similar to the environment that we use internally. I booted the os, installed a few of my favorite tools (screen, git, etc), installed dependencies, and started installing Ibis using our standard procedure.  The first build failed rather quickly – mostly because of differences in locations of files in Raspbian compared to our normal environment. After small number  of iterations finding these files and installing them I started what I thought would be ‘the’ build. After some time of looking like an easy and successful build, gcc barfed and the build failed with:

error: command 'gcc-4.6.3' failed with exit status 3"

This was new to me – and a quick search of a number of C programming books in our library and some web searches turned up nothing!

The Pi has 256MB of memory and it is divided roughly 40/60 between video buffer and system memory. I had intentionally tried the build out of the box with no memory reallocation and though gcc left no real information as to why it failed, I was sure that low memory was the cause. I reassigned as much RAM away from the video buffer as I could (I am only using terminal sessions after all) and tried again. The build failed again, gcc barfed again, but this time farther along. At this point I decided to give the Pi gobs of memory by attaching an 8GB USB stick so that it has 2GB swap and the remaining 6GB as /home. Here is what the configuration looks like on my desk:

The next build completed successfully! The capacity of this configuration is extremely modest. Two simultaneous sessions work, and the math isn’t optimistic about many more, but for $35 (plus a spare usb stick) and less than 1 watt of power, this is a true general purpose application which hints at the potential of small, affordable, and modestly powered computers.

I’m thinking the next experiment might be making a solar power system to run a cluster of these 24/7 around the clock. Or maybe a DIY S88/S95 compatible process control system with the Pi supervisory control units and Arduino loop-controllers. Or when my 16 core $99 Parallela board gets here I’ll make a solar powered Software Defined Radio, data telemetry, and epub serving robot to roam the seas.

Why not?    :-)

Check these out:

Getting Started with Raspberry Pi

Raspberry Pi at the Maker Shed

(12-12-12 edit:  - link to http://pi.ibisreader.safarilab.com:9021 removed, as this cool little board is moving on to it’s next project. Stay tuned!  ch)