And Now For Something Completely Different

Friday was my last day working in Ann Arbor (1hr away, by car, 4 by train/bike) at Greenview Data. At Greenview I had helped to build an email archiving system, writing mainly Ruby and Perl code using Agile (XP) development practices. Two of the best years of my career were spent there – I learned so much and discovered who I was as a software craftsman. My coworkers there helped keep me accountable and we had a great time learning together new concepts and some neat tools. In the end, we have an email archiving system that provides discoverability and redundancy across 3 data centers in the US.

The biggest change in the last year has been my migration from graphic IDEs and text editors to vim. Once I discovered it’s underlying philosophy, I was hooked. Now Linux has returned as my development platform of choice. Linux especially works well when working with Ruby C extensions.

This next week I’ll be finishing some work for the State of Michigan and then February begins my work at BrighterPlanet! Conservation, sustainability, and environmental stewardship have become hot issues for me over the past three years. I’m really excited to be working on that front as a Rails developer developing apps that help people start thinking about their relationship with the environment and helping some grassroots environmental projects get started through the company’s project fund. When I found BrighterPlanet’s open telecommuting position, I couldn’t pass it up.

So now I will be working from home during the day. Julie and I have been setting up the office as our coworking facility and I’ve been tinkering around with my old PC to see if it’s usable as a Linux development platform. I also got a nice 22″ Dell monitor (same components as Apple LCDs) to reduce neck straining with my laptop. It also doubles as our TV

2010 is already shaping up to be an excellent year!

Getting SPSS Statistics 17 to Work with Mac OS X Snow Leopard

My wife is a grad student who does quantitative studies from time to time. Her advisor recommended that she use SPSS to help her generate statistics from her research data. Of course, SPSS is a gigantic, bloated Java app developed by IBM. Being an IBM product targeted toward large institutions, it also costs an arm and a leg.  I really wish R was more well known and widely used because it is free software.

My wife recently found that after I upgraded her laptop to Snow Leopard, it rendered SPSS unusable. Mac OS reported that the application is incompatible with Snow Leopard. For a brief time, SPSS offered a free patch that allows you to use SPSS with Snow Leopard, but it expired December 31st, 2009 and you are now told to buy an upgrade for $100.  A poster mentioned in a help forum that the only reason SPSS 17 didn’t work with Snow Leopard was that SPSS 17 was designed for Java 1.5 and Snow Leopard only comes with Java 1.6 installed.  My blood boiled as I realized that SPSS was attempting to charge me $100 for not having an older version of Java installed. I was also suspicious of the fact that Mac OS warned of an incompatibility before the application even attempted to launch. How was it so sure?

I found some tutorials for getting SPSS 17 to work with Snow Leopard, but they involved magic and cargo culting.  Therefore, I give you the simplest instructions for installing it:

1. Download a Java 1.5 package that was designed for Leopard, but will work with Snow Leopard, too.
2. Extract the package by double-clicking the downloaded file.  A “1.5.0″ folder will appear.
3. Drag the 1.5.0 folder into Macintosh HD/Library/System/Frameworks/JavaVM.framework/versions. Say yes you want to overwrite and authenticate with your user name and password.
4. In spotlight (click the magnifying glass in the top right) type “Java Preferences” and click the top hit.
5. In the bottom-most box, drag Java 1.5 32-bit to the top of the box, ahead of Java 1.6.  This tells OS X to try and use Java 1.5 to open your apps before it tries the newer 1.6 version.
6. Download and install the SPSS 17.0.2 patch
7. This is the ridiculous part. Snow Leopard ships with a black list of applications that it “knows” are incompatible. Apple is trying to protect us, here. The only way it knows you’re trying to run an incompatible application is that it checks the name of the application you try to run against its blacklist. Simply rename your SPSS program by going to Applications/SPSSInc/Statistics17, highlight SPSSStatistics17.0, press return, then rename it to something like SPSS. Now Mac OS will not recognize that you’re running SPSS 17.

Happy statisticing, or whatever it is you do…

Ruby Development on Linux, Part 3

I’ve posted earlier about my attempts to find the perfect development environment on Linux, an environment that would match the elegance and ease of TextMate on OS X.  I’ve always been a fan of the freedom that you get with Linux and I’ve been willing to sacrifice prettiness as long as the environment is intuitive and easy to use.

So far, my journey has been wrought with failures, but there have been a few bright spots.

By far, my favorite platform is Ubuntu.  They’ve really nailed the desktop usability aspect as it trumped CentOS and Fedora in being able to recognize my video card and use dual monitors.  However, Ubuntu still suffers from some quirks.  With full visual effects enabled, I can only open so many instances of Firefox without strange things like page content not scrolling.  When I hit a certain number of windows open over several desktops, Gnome freaks out and merges them all into a single workspace, disabling full visual effects.

I have tried many text editors and IDEs but all come up short in some way.

I can’t stand vi’s separate edit and navigation modes.

XEmacs is quick and powerful, but there are too many quirks that keep me from loving it.  The speedbar widget exists as a separate window, which makes task switching more of a pain.  Simply finding settings and saving them across sessions is cryptic and difficult.  Installing a language extension requires some elisp programming.  I just want to click!  Copy and paste are broken.  If I copy in emacs, I can’t paste into another app, and vice-versa.  The only way to do it is to use the edit drop-down menu in each app.  Aside from that, the actual act of text editing in emacs is a breeze.  However, I still miss the Textmate macros that generate things like RSpec describe and it blocks.  One redeeming quality of xemacs is that I can have multiple windows open.

Netbeans works, but is a bit slow.  I also hate having to create an nbproject folder to keep project state.  The same goes with Eclipse, Aptana, JetBrains, and all the rest.  IDEs are too cumbersome, especially when most of them force you into a MDI interface.  I really like having many different windows open when editing a project.  Tabs just don’t do it for me.  In fact, that’s one area where Textmate disappoints me.

I had tried to make gedit act like Textmate, but it would crash a lot.

Lately, I’ve had the best results with Komodo Edit.  I’ve been able to customize the keyboard shortcuts so that alt acts just like the command button in Textmate/OS X.  It’s very helpful when I have to switch off our Mac onto the Linux box.  I’ve also set up a “cobalt” theme.  Working in split-screen mode with the whole project window spanned across monitors works well enough for my desire for multiple windows.  I can have the code on one side with the spec on the other.  Komodo also features the quick file open command and can show you where a method is defined.  The drop-down autocomplete is also nice.  There is no integration with rspec, et al, but I’m fine using the terminal.  In fact one annoyance I have with Textmate is that any output to STDOUT in a spec is assumed to be HTML in Textmate’s output screen.  This means I have to use HTMLEntities.new.encode when inspecting an object.  With the terminal, text is text!  Komodo also requires project files to be maintained in the root directory of your project, but I guess I can live with that.  One final complaint: Komodo tends to crash if I have too many separate windows open.  I guess I just can’t have 6 projects open at a time.

Redcar is a very promising project.  It’s designed to act just like Textmate and will support TM bundles.  It’s also written in Ruby!  The only thing that has kept me from using it is that the gtk-ruby libraries that it uses have C extensions that are written specifically for Ruby 1.8.  I’ve spent a good chunk of time porting them to 1.9 compatibility, but it’s such a large task I don’t have time to port it all.  I do have Ruby 1.8 installed in parallel, but trying to get everything to run proves difficult when different components of Redcar try to run /usr/local/bin/ruby (which points to 1.9 on my system).

Over all, Komodo on Ubuntu has made life manageable, but not quite the experience I get with Textmate.  I’m sure once Redcar is 1.9 compatible I’ll be switching.

CurbFu – Now With Killer Testing!

Matt Wilson and I have been making some awesome new updates to CurbFu – our convenient wrapper for Ruby’s curb gem – itself an wrapper around libcurl.

Our latest improvement revolves around integration testing. At Greenview Data, we have several Rack apps that talk to each other, including a Rails app. While all of code for the individual apps was unit tested, we still lacked a good integration test to make sure everything played well together.

In the past I had tried to set up a VM instance that was controlled by a series of RSpec Story Runner tests. Getting the test server instances updated and running was a pain, managing all of the running processes was a pain (especially testing a system with down processes), and writing tests to verify data became quite contrived.

With the introduction of Bryan Helmkamp’s Rack::Test and Rails 2.3’s support for Rack, I dreamed of having these apps running in memory, able to send HTTP requests to each other without having to set up a separate integration testing system. Now that all of our apps use CurbFu to send HTTP requests to each other, we seized the opportunity to create a testing module that override’s CurbFu’s get/post/put/delete operations. Instead of sending a real HTTP request over the internet, CurbFu will now call any rack apps you specify directly using Rack::Test.

Essentially, this system is similar to FakeWeb and Webrat’s newly added Rack support, but the difference is that if you use CurbFu for all of your HTTP communication, you can quickly and easily set up an integration test environment or set up smart mocks of HTTP services.

How does it work?

require ‘curb-fu’

CurbFu.stubs = {

‘a.example.com’ => RackApp1.new,

‘b.example.com’ => RackApp2.new

}

class RackApp1

def call(env)

CurbFu.get(‘http://b.example.com’)

end

end

class RackApp2

def call(env)

puts “WOW!”

end

end

CurbFu.get(‘a.example.com’)

#=> WOW!

require 'curb-fu'

CurbFu.stubs = {
  'a.example.com' => RackApp1.new,
  'b.example.com' => RackApp2.new
}

class RackApp1
  def call(env)
    response = Rack::Response.new
    response.status, response.body = [200, '']
    CurbFu.get('http://b.example.com')
    response.finish
  end
end

class RackApp2
  def call(env)
    response = Rack::Response.new
    response.status, response.body = [200, "WOW!"]
    response.finish
  end
end

puts CurbFu.get('a.example.com').body
#=> WOW!

Loading a Rails app is a bit more involved:

require 'curb-fu'
require '/path/to/rails/app/config/environment'
require 'dispatch'

CurbFu.stubs = {
  'my.cool.railsa.pp' => ActionController::Dispatcher.new
}

We’ve set up a series of Cucumber stories to test the interplay between our various rack apps. By having the rails app in memory, we’re able to access ActiveRecord objects from our step definitions, as if we’re within the rails app. We’re also able to stub specific methods on objects within our rack apps and we throw up quick little Rack apps that can pretend to be things like a Solr server or a site hosting an RSS feed. The sky is the limit!

Once caveat we’ve run into is the issue of namespacing. It may be a good idea to namespace your Rails model classes as they may conflict with other loaded Rack apps.

Have fun testing!

Links:
CurbFu

Double Negatives

Recently I was working on a project adding new functionality that would queue a long-running task to a background job. The original code looked something like this:

receive_data
forward_data

My task was to run enqueue_data instead of forward_data since the forwarding was taking too long. I wanted to retain the ability to run forward_data if a specific parameter was passed in. Since I was creating a parameter that would override a default feature, my first intuition was to name this parameter “noenqueue”:

receive_data
if params['noenqueue']
  forward_data
else
  enqueue_data
end

I then smacked myself in the forehead when I realized I had “pulled a Microsoft.” I had created a setting that is named such that setting it to true would disable a feature, while setting it to false would enable the feature. You see this all the time in Microsoft products.

It irritates me because you have to think in double-negative terms when setting the parameter. Instead, it is much more intuitive to create settings that are positive – that setting them to true will enable something.

Lesson learned: I renamed the parameter to “forward_immediately”:

receive_data
if params['forward_immediately']
  forward_data
else
  enqueue_data
end

One good syntactic feature of Ruby is the unless keyword. I prefer to use “unless variable.nil?” instead of “if variable” if I want to only operate on non-nil objects. I find it a bit more intuitive.

unless document.nil?
  document.write
end

However, you can still get into trouble when you need to branch:

unless document.nil?
  document.write
else
  document.new(str).write
end

An else branch on an unless is, in my opinion, the least readable or intuitive way to write branches.

Great Lakes Ruby Bash 2008

I’m really excited about the upcoming Great Lakes Ruby Bash in Ann Arbor on October 11th.  The distinguished Rubyists of the Grand Rapids Ruby Users Group are joining forces with us (Ann Arbor Ruby group) to put the conference together.  After attending eRubyCon in Columbus last weekend, I have to say the local conferences are much more engaging than the mega-conferences like RailsConf.  It’s great being able to get a chance to talk to everyone.  Hope to see you there!

BackgrounDRb 1.0.4 – Considerations

Today I gleefully updated BackgrounDRb to 1.0.4 after I learned that a new release was out that supported clustered BDRb servers.  I quickly learned that regiester_status has been removed.  That’s right.  All that code that sends data from your workers to your app need to be refactored to put data into cache[job_key].  This approach is more thread-safe, however.  Perhaps you could even write your own wrapper that puts data into cache on a call to register_status.

All in all, however, there are some really cool new features related to clustering.  I was starting to think for a while that BDRb had been abandoned and that Bj was the new way to go.  This release clearly puts BackgrounDRb on top for spawning asynchronous tasks.

Does MobileMe’s Push Email Really “Push?”

When Steve Jobs first announced “push email” for the upcoming iPhone 2.0 software, I was skeptical.  The two protocols I knew of, POP3 and IMAP, operate on a protocol that is always initiated by the client.  The user has to continuously check in with the server to see if new messages have arrived.  With POP3 and IMAP, you can approximate push email by polling server several times a minute.  I figured that Exchange supported some sort of persistent connection that allowed push, but from the way it was presented at Steve’s keynote, it seemed that they were touting push email for any mail account accessed by an iPhone.  I wrote off “push email” as a marketing ploy.  Im my mind, the mail server would have to know where the client was at all times to “push” updates.

Well, it turns out that MobileMe supports Push-IMAP, a relatively new protocol.  The way it works is a client logs into the MobileMe server with a long-lived HTTPS connection.  Through this connection, the client sends any updates to the server.  The server sets up a long-lived response to send back notifications of new messages.  Data is compressed to help lower bandwidth requirements.  P-IMAP is an open protocol and a P-IMAP to IMAP bridge can be set up to allow legacy mail servers to provide push services.  In addition to email updates, P-IMAP allows other data, such as contacts and calendars to be synchronized.

What seems surprising to me is that Apple is going to have to set up a system that will have to handle millions (well, at least thousands) of simultaneous connections.  Let’s hope they can do it!

So, is MobileMe’s push really “push?”  Well, technically, no.  The client still has to initiate and maintain a connection to the server.  If the server kept track of what IP address each iPhone had and initiated connections out to each iPhone, it would be a true push system.  In reality it’s a more scalable way to handle continuous polling.

 

More information:

P-IMAP

How I Got Started Programming

Continuing the meme:

How old were you when you started programming?

Officially, it was my freshman year of high school.  Unofficially, the most “programming” I did was modifying config files to get my PC games to run faster on our Gateway 386.  A favorite game of mine growing up was “Rocky’s Boots” on our Tandy 286 which involved completing electronic circuit puzzles.  

 

 

How did you get started in programming?

I liked computers and my high school just happened to offer a BASIC programming class on old (even then) Apple //e’s.  After that I was hooked.  I think I inherited my dad’s creativity – his outlet is woodworking and construction.  I love to build things and doing it on a computer was the most logical step, since that’s where I spent all my time anyway.

My first programming job was a part-time gig during high school at an ad agency creating Access database applications for telemarketers’ data entry.

What was your first language?

Good old BASIC.  I quickly learned the evils of spaghetti code.  My final project for the BASIC class in high school was to do a “hi res” (80×48) animation.  It featured a bunny jumping through a static forest scene, a gun popping out from the side of the screen, shooting the bunny in the head.  Perhaps I was playing a bit too much Wolfenstein 3D?

Later on in high school, I moved through Pascal, C, and C++.  A past-time of mine in the computer lab was to create Myst-style dungeon crawlers using Hypercard.  There was always a secret spot to click to defeat the boss.  

What was the first real program you wrote?

How do you define “real?”  My first for-money program was a simple Access database for telemarketers.  Of course, there is a minimal amount of programming involved there.  My college classes offered me plenty of interesting projects.  One was a series of programs written to compute large matrix multiplications over distributed systems using various threading/multiprocessing libraries.  A part-time job I had during college consisted of small web apps (mainly shopping carts, CRM) written in ColdFusion.  

What languages have you used since you started programming?

BASIC, Pascal, C, C++, Scheme, Java, JavaScript, ObjectiveC, AppleScript, ColdFusion, T-SQL, PHP, C#, Visual Basic, Perl, and last, but not least, Ruby!

By far, Ruby and Scheme have been my favorites.  Ruby’s flexibility, friendliness, and the way it so nicely complements TDD/BDD have won me over.  While I’ve never “monetized” my knowledge of Scheme, it was so much fun to think “functionally” to solve a problem.  

What was your first professional programming gig?

Working for a contractor in Northern Virginia.  I worked on some C/C++ interfaces for USPS mail sorting machines and a system for nursing assistants to document nursing home care that involved a ColdFusion web app and a VoIP interface to an interactive voice system.  I learned quite a bit about the “real world” of programming.  At that time I was into creating “sugar castles” of code in my head and spent a lot of time fixing mistakes I thought I was incapable of making.  It was a good reality check.

If there is one thing you learned along the way that you would tell new developers, what would it be?

A multipart answer:

1. Don’t typecast yourself into knowing one specific language or tool.  Chances are it will become obsolete and you will be “that guy” who maintains that dinosaur system and is simply there to punch the time clock and get a paycheck.  The world of computer science and software development is so much bigger and more interesting than just Java or Ruby or MVC or TDD.
2. Be open.  Admit when you’re wrong.  Learn and improve.
3. Get things done.  Wasting your time over-engineering a system will be simply that – a waste of time.  Your customers don’t care how cool your algorithms or languages are and you’re better off spending that time testing and refining the user experience – delivering what the customer actually wants.

What’s the most fun you’ve ever had… programming?

My languages course in college – writing Scheme code.  Again, the functional paradigm is a lot of fun.  Hacking with others on fun side projects over pizza and beer is always a blast.  It’s the geek’s version of building a shed in your backyard with your buddies or jamming together in your basement.

Linux Editor Update

Well, no sooner than I had written the previous article, did I consider NetBeans.  At this point I was willing to try anything, so I downloaded the special Ruby distribution and gave it a whirl.  One of the prereqs was the Java 1.6 runtime, a hefty download.  As NetBeans opened for the first time, I was greeted with a splash screen with a progress bar inching along as various Java libraries were loaded.  This did not bode well!  My bloat-meter was starting to register.  

However, once NetBeans was up and running, I found it to be pretty intuitive and it runs very well.  Response to input is instantaneous and the editor does a good job of not getting in the way.  I really enjoy the SVN integration as well.  It’s very easy to ignore files and perform the usual updates/commits.  The merging tool has some of the slickness of Apple’s FileMerge and many of the features of my favorite diff tool, Beyond Compare.

My only beef so far is that auto-complete is invoked with CTRL+space and I don’t think you can map it to TAB and still be able to use TAB for its original purpose.  The autocomplete also only works by selecting items from a drop-down.  It won’t just guess what you want.  

But overall, NetBeans has been my favorite Linux text editor for Rails.  However, I think I’ll have to stick with gedit or Eclipse for my Perl editing.