Industrialization and Web Development

On my way down to Illinois today, I listened to a podcast giving an overview of American history before 1870.  In it, Dr. Reilly talks about the industrial revolution.  I was enthralled as I contemplated the changes that the revolution brought about and the similarities to the changes taking place in web development.

In the 1800’s manufacturing moved from a guild system toward the use of unskilled labor and interchangeable parts.  Under the guild system, an apprentice would live and work for a master tradesman who provided food, shelter, and training.  Goods
were produced from start to finish by a single tradesman. Volume of production was limited and a great deal of skill was required of the tradesmen, who needed to have a holistic understanding of the products they produced.

The guild system remained primarily because it was most economical to have a local tradesman produce goods.  It was too costly to import goods from across a continent.  Once canals and railroads provided cheap freight services, it became economical to produce a high volume of goods and ship them to distant customers.  Thus, the need for mass production arose.  Instead of hiring ten times more tradesman to produce ten times more products, it made more sense to hire unskilled workers to perform specialized tasks, thus “pipelining” the production process.  Tradesman began to take more managerial roles, training the unskilled laborers in their specific jobs.

At the same time, the idea of using interchangeable parts arose.  With interchangeable parts, it was much easier to repair and maintain products.  Before the revolution, a product was custom made and its design was known intimately by the tradesman who made it.  If it needed to be fixed, the best person to fix it was the one who made it.  With interchangeable parts, a product’s parts could be easily replaced if broken and could be repaired or maintained by anyone familiar with the industry.  Furthermore, a factory producing widgets can make the widget almost completely out of interchangeable parts produced by other companies.  No specialized knowledge of how each part is made is required.  The factory can simply make new widgets out of preexisting components.

The industrial revolution was a drastic change for the tradesman who were accustomed to the old guild system.  The tradesman who prided themselves on quality (and high wages for specialized work) strongly opposed the revolution.  They considered the mass-produced goods to be inferior to their custom, hand-made products.  They formed unions to try and preserve their wages and job security. But ultimately, mass production won out and in many industries completely replaced the guild system.

So how does this relate to web development?  In many ways there has been an “industrialization” of how web applications are developed.  In many cases today, web applications are (or have been) written by one or a few programmers who are intimately familiar with their custom implementation.  If a programmer leaves, the web development company is in a bit of bind because knowledge of all of the intimate details of the system is lost and a great deal of time is required for new programmers to wrap their heads around the system in order to maintain or improve it.  Over the past seven or so years, there has been a trend toward specialization of work and an increase in the use of libraries, code generators, and frameworks, similar to the idea of interchangeable parts. 

Prior to our current days of web 2.0, the only division of labor that normally existed was between graphic design and programming.  As web applications have become more complex, many companies have moved from hiring jack-of-all-trades web developers to hiring specialized Flash developers, UI developers, database developers, and server-side develoeprs.  Companies seem to care more about whether a programmer is intimately familiar with J2EE or DOM scripting rather than a holistic understanding of web applications and computer science.  To run with the industrial revolution analogy, it’s like an employer who is more interested in how well a worker can operate a welder, rather than being able to build an entire car by hand.

Libraries, frameworks, code generators, web services, and mashups can be considered the “interchangeable parts” of the web.  As long as an unskilled programmer knows a specific framework, say CakePHP, he can jump into any CakePHP project and easily maintain it. Not as much specialized knowledge about the application is needed compared to a custom-written, classic PHP site.

While there are similarities between the effects that the industrial revolution had on industries like blacksmithing, shoe making, and automobile making, and the effects of industrialization on web development, there are some differences.  While the industrial revolution nearly completely replaced tradesmen with unskilled workers (and even robotics) in industries where simple products are produced, it has not replaced tradesman in every industry.  For example, in most cases skilled contractors are still needed to build houses and large buildings.

As I see it, there are currently three types of web applications being produced today.  The first type is the static, informational web site.  There’s not much for a programmer to do here.  Most of the work is for the designer.  At one time, someone who knew HTML was needed, but apps like Dreamweaver have replaced that need.  The second type is the CMS-style site that might have news, events, subscriptions, or blogs.  These sites used to require dedicated developers, but now packages like Drupal, PHPNuke, and wikis have replaced this need.  Some companies still develop these type of sites from scratch, but as customers become more educated and competitors outsell the custom shops with prepackaged CMS systems, the custom-designed CMS site is a dying breed.  The third type of web app either has complex business rules or it stands on the bleeding edge of technology.  These type of apps will need developers, but not necessarily the “jack of all trades” developer.

On the one hand, it is comforting to know that, at least for the time being, there is a need for skilled developers.  The web development guild system hasn’t yet been eliminated by unskilled labor.  However, it is a bit disturbing to see companies move toward hiring specialized developers who are “intimately familiar” with specific languages or platforms, rather than
developers who have a holistic understanding of web applications.  In addition, many companies are hiring programmers with little skills to churn out sloppily-coded applications.  What is a “craftsman developer” to do? 

Continuations in Ruby

A week ago, Dr. Frens spoke about continuations in Ruby at the Michigan Ruby Users Group. It was very interesting as it brought back my college days when I learned Scheme and functional programming concepts. I always enjoyed functional programming because it presents such a different way of thinking through problems.

I was going to blog about his presentation, but it turns out he blogged all his own notes already. But what I found interesting is the sort of hybrid nature that Ruby has. It is both an object-oriented language and it also supports some functional programming concepts. The drawback Dr. Frens found was that Ruby doesn’t do tail recursion optimization, so it’s not really worth using continuation passing style functions. Interesting, nonetheless!

Video on the Web

It’s really great that video is becoming more prevalent on the web. We can now ditch our TVs and watch video online. We can watch presentations given at conferences that we missed or couldn’t attend.

However, it’s really annoying when all I want is some quick information, but I have to sit or scrub through 5 minutes of irrelevant babble to get to the gold. For instance, when Android came out, Google posted some nice, long videos about their technology. If you wanted to know any detailed information about it, you had to sit through the boring video.

Technocracy Radio discussed fora.tv in their last podcast. Basically, it’s sort of the C-SPAN of the internet, but not just restricted to politics. While having all of this video is great, I think having a textual transcription of the speech is much more valuable. You can easily search text and do all kinds of data mining with it. I think the next breakthrough in media is going to be when speech to text recognition is perfected, fast, and cheap. This way, we can search for any time the President says, “nukular.” We can also have TVs that recognize a viewer’s laughter and flag the funniest quotes in a Simpsons episode. Awesome!

Eager Loading

One thing that made me uneasy about Rails’ ActiveRecord was how it handles associations. For instance, say I have a table of products in the database and another table listing comments about the products. So, for each product we have many comments.

Back in the good ol’ ColdFusion days, I would write a query that joins the two tables. Then I would loop through each field and have some funky conditional statements, like the following:

<cfset prevProductId = 0 />
<cfloop query="qryProductsWithComments">
 <cfif product_id neq prevProductId>
  ...if this is not the first product, finish printing the previous product's template...
  ...we've hit either the first product or a new product, so print out the beginning of the product template and the first comment...
 <cfelse>
  ...this is another comment, so print out just another comment...
 </cfif>
 <cfset prevProductId = product_id />
</cfloop>

That’s pretty darn confusing, but it was better, performance-wise, than executing a new query every time you iterate over the products loop, like so:

<cfloop query="qryProducts">
 ...spit out the product template...
 <cfquery name="qryComments" ...>
  SELECT * FROM comments WHERE product_id = #qryProducts.product_id#
 </cfquery>
 <cfloop query="qryComments">
  ...spit out a comment...
 </cfloop>
</cfloop>


By default, ActiveRecord will perform similarly to the latter example. However, with eager loading, you can get the same performance as the former example without all of the confusing if statements:


# in our model:
class Product < ActiveRecord::Base
 has_many :comments
end


# in our controller:
class ProductsController < ApplicationController
 def index
  @products = Product.find(:all, :include => :comments)
 end
end


# in our view:
<% for product in @products -%>
 <%= render :partial => "products/product" %>
 <% for comment in @products.comments -%>
  <%= render :partial => "comments/comment" %>
 <% end -%>
<% end -%>

So much cleaner! And easier!

With ASP.NET 2.0, you’re pretty much stuck doing the equivalent of the second CF example using the server controls or you have to build your own object data source that will work similarly to ActiveRecord, but you’ve just spent an hour or so reinventing the wheel.