Modern Modularity & Iterative Development

I recently wrote an app in about two hours: a highly tailored, very powerful tool for a clerk. And I did it as a Web CGI app using straight Perl without the benefit of a rapid application development environment or even a framework like Mason.

Hurray for me. Except, of course, I didn't really do that. It is true that I wrote the app in about 45 minutes and then spent a little over an hour in a tight user feedback / iterative development cycle, resulting in the finished product. But this is the movie version, the marketing hype, the storyline that ignores all the boring ground work that went before.

It would be more accurate to say that I spent about dozen hours over several weeks on various stages of this project: still fast, still cheap, just not spectacular (or utterly unbelievable).

From my perspective, the process went like this:

1. The client defined a problem: clean up a server's disk area by removing redundant files stored by a system of ours.
   
2. I defined the reconciliation algorithm, using md5sum to confirm that files exist in both places. Along the way we encountered some horrific special cases, but that is par for the course in the real world.
   
3. I wrote a audit program to put a random sample of these files in a web page for review; our clerk reviewed several batches of these files spread out over the entire repository. We found that the issues were highly concentrated in time windows, so we tailored our approaches to fit the eras that we found.
   
4. Based on her feedback, I reused much of the audit program to create a web-based UI to support her review and I wrote a couple of small helper utilities to automate the review process that the clerk had used. She confirmed her earlier era-based findings.
   
5. I added a database table to hold the information that the clerk was tracking on her paper pad and which she had figured out was what she needed to know. This allowed us to break down the repository into manageable chunks and to start tracking the clerk's progress more accurately.
   
6. I wrote a daemon to do prep work in the background and another one to automate as much of her review as I could. This cut the number of items requiring human review by 90% and allowed us to start automatically correcting that 90% while she worked on the ugly 10%.
   
7. Once all that was in place, I wrote the final app, a UI to review the contents of the database as set up the by daemons and to add editing functionality to support saving the results of the review. Then I added a few links to those helper apps to support the reviewing and I was done.

This is how I write good user-intensive software: a design phase which moves quickly to practical applications; we figure out the ideal process and the software trails quickly behind until we have a system that we all trust and that does the job.

Note that my utilities and previous code are highly reusable in both senses:

• I can use the code with little or no modification in different apps, which means that I only have to develop and validate once but use many times.
  
• The stand-alone web helper apps are easy to string together to enrich the environment provided by the main app. This is an homage to the Unix "string of pearl" philosophy which I find works so well at the O/S level.

In theory I should get both of these benefits from the Object Oriented programming paradigm, but in practice I don't seem to.

In this day and age, the web technology allows us to have separate-but-easily-integrated pieces and modern programming languages allow us to reuse chunks of code easily so there is no excuse for not producing polished and power apps rapidly and cheaply.