Modern Modularity & Iterative Development

I recently wrote an app in about two hours: a highly tailored, very powerful tool for a clerk. And I did it as a Web CGI app using straight Perl without the benefit of a rapid application development environment or even a framework like Mason.

Hurray for me. Except, of course, I didn't really do that. It is true that I wrote the app in about 45 minutes and then spent a little over an hour in a tight user feedback / iterative development cycle, resulting in the finished product. But this is the movie version, the marketing hype, the storyline that ignores all the boring ground work that went before.

It would be more accurate to say that I spent about dozen hours over several weeks on various stages of this project: still fast, still cheap, just not spectacular (or utterly unbelievable).

From my perspective, the process went like this:

1. The client defined a problem: clean up a server's disk area by removing redundant files stored by a system of ours.
   
2. I defined the reconciliation algorithm, using md5sum to confirm that files exist in both places. Along the way we encountered some horrific special cases, but that is par for the course in the real world.
   
3. I wrote a audit program to put a random sample of these files in a web page for review; our clerk reviewed several batches of these files spread out over the entire repository. We found that the issues were highly concentrated in time windows, so we tailored our approaches to fit the eras that we found.
   
4. Based on her feedback, I reused much of the audit program to create a web-based UI to support her review and I wrote a couple of small helper utilities to automate the review process that the clerk had used. She confirmed her earlier era-based findings.
   
5. I added a database table to hold the information that the clerk was tracking on her paper pad and which she had figured out was what she needed to know. This allowed us to break down the repository into manageable chunks and to start tracking the clerk's progress more accurately.
   
6. I wrote a daemon to do prep work in the background and another one to automate as much of her review as I could. This cut the number of items requiring human review by 90% and allowed us to start automatically correcting that 90% while she worked on the ugly 10%.
   
7. Once all that was in place, I wrote the final app, a UI to review the contents of the database as set up the by daemons and to add editing functionality to support saving the results of the review. Then I added a few links to those helper apps to support the reviewing and I was done.

This is how I write good user-intensive software: a design phase which moves quickly to practical applications; we figure out the ideal process and the software trails quickly behind until we have a system that we all trust and that does the job.

Note that my utilities and previous code are highly reusable in both senses:

• I can use the code with little or no modification in different apps, which means that I only have to develop and validate once but use many times.
  
• The stand-alone web helper apps are easy to string together to enrich the environment provided by the main app. This is an homage to the Unix "string of pearl" philosophy which I find works so well at the O/S level.

In theory I should get both of these benefits from the Object Oriented programming paradigm, but in practice I don't seem to.

In this day and age, the web technology allows us to have separate-but-easily-integrated pieces and modern programming languages allow us to reuse chunks of code easily so there is no excuse for not producing polished and power apps rapidly and cheaply.

When You Can't Read The Code

As the large enterprise solution becomes more and more common, I find that the keepers of such systems seem less and less interested in putting in the time to become domain experts.

As IT systems consultants, we are often interacting with these enterprise solutions. Some percentage of the time, we encounter behaviors in these systems which we find unexpected or undesirable. These behaviors may or may not be bugs, but we feel that we should understand them before we ignore them or program around them.

Back in the day, when more apps were homegrown and more systems were provided with source code licenses, the people we asked about unexpected or undesired behavior were the client's own technical people. If they did not know the answer, then they could find out by reading the code.

Increasingly, we find that our client contacts are either non-technical people or out-of-date technical people. Only a small minority of contacts are technically competent, helpful providers of deep background and technical information. We often have to bypass the client and go straight to their large system vendors because there is no local technical expertise.


To our dismay, it is our technical bretheren, our peeps, who are the hardest to deal with: now that they cannot read the code, they tend to either shrug off our questions or give us shallow guesses off the top of their heads. After all, if you can't be a hard master then why not just give up?

To our delighted surprise, the non-technical people are generally easier to deal with because they know that they don't know and they look for the answer. They will actually contact the vendor for information, or find documentation, or give us the benefit of the user's experience.

While we don't like treating systems as black boxes or as puzzles into which we pour various inputs and monitor the outputs, at least these methods actually turn up results. Getting half-baked guesses from formerly expert techies just wastes our time as we discover just how half-baked the guesses are.

Just recently I spent an afternoon running a series of experiments as we tried various inputs into a large system to see if we could figure out how to accomplish a particular upgrade for the user. We started with a half-baked guess from a supposed expert techie, proved that this techie was wrong about how the large system worked and got down to the business of figuring out to do what must be done.

It seems to me that technical people have to be able to operate in the absence of hard data. My checklist for answering technical questions goes like this:

1. Query experts if you can
2. Read the code if possible
3. Read the technical documentation if there is any
4. Query super-users if you can
5. Read the user documentation if you can
6. Query experienced normal users if you can
7. Write code to grok databases, logs, etc
8. Try to reason from first principles
9. Guess because there is absolutely no other option

And remember that when talking about technical matters, which are not generally matters of taste or subjective opinion, trust but verify.

VMs and O/Ses and Bears, Oh My!

IT Nirvana?

Computer systems offer more bang for the buck than ever before. There are more and better options than ever before. Storage is cheap and plentiful and processors are mighty and don't catch fire when you use them.

In theory, the size of data set or processing problem that is now "easy" should be so big that just about anything I do should be easy. In fact, since there this so much computing resource available to help me do it, just about anything I want to do should be simple as well. But I am finding that this implicit promise of simplicity is not being fulfilled, at least not at the system level.


I Am Not Immune

Consider my humble desktop at work. It is an Ubuntu box which is backed up automatically and offsite by the mighty boxbackup utility. Thanks to Virtualbox, I use virtual machines, of various types: a windows7 VM for development and to host my iPhones; various Linux VMs for various special purposes.

Recently we lost power while I was out of the office. My trusty desktop shut down gracefully because I run apcupsd. Hurray! apcupsd even told me when the power went off, although in true open source geek fashion, it reported the time in GMT.

However, the VMs did not shut down gracefully when their host did. Apparently, this is something that I have to set up myself using launchd. One of the VMs was trashed when the host shutdown; the other two were fine.

So on to restoring the trashed VM: the good news is that I found that my boxbackup repository was up to date; the bad news is that it contained a copy of the crashed VM in it. I am a belt-and-suspenders kind of guy, so I have a manual local back up to check: yes, I have an image in the on-site backup which is a few months old, but that is ok: these VMs do not change much over time. So, problem solved in the short-term but not the long-term.

In the long-term, I need to get some professional sys admin time or I need to change into my super tech costume and chase down these issues myself. I strongly favor the first option: the more I know, the more I value the knowledge of others. There is also the factor of money, though: the longer the Great Recession goes on, the less inclined I am to shell out real money unless I have to.


The 21st Century Data Center So Far: Boo

I bring up the plight of my desktop to emphasize that the rant that follows is  not simply screed against any particular sys admin but an observation about the environment in which most sys admins have to do their jobs.

As we struggle to deliver software and service on our client's hardware we constantly run into misconfiguration of hardware, virtual hardware, operating system software and services such as web servers and database servers.

We also run into poorly implemented policy and self-contradictory policy which doesn't help and somehow offends me more: can't we at least agree on a usable definition of what we are trying to do?

As VMs become more and more common, and the ability to deploy them correctly more and more rare, we are being forced to return to the "my software, my hardware, my responsibility" model of the past. Especially when we find that off-brand Unix distributions such as AIX seem to lag so woefully behind current.

In the 1980s and 1990s, we used to drop "departmental servers" into our client's work areas because the company mainframe was too expensive, too dedicated and too central to use. In what is now known as the Apple model, life was good: the client had one contact point and we had a known, stable environment.

In the 2000s, we tried to get with the program and use existing infrastructure such as database servers, DNS and DHCP . This was a nightmare: for one thing, every new IT administration seemed to want to do things differently: Microsoft! Open Source! Back to Microsoft, but maybe running some Open Source software on the windows server! Ok, how about thin clients which were sort of Windows? Oh, were you still using the old DNS? Sorry about that--wait, let's use Active Directory for authentication! Is it set up correctly? Who knows!


A Computer System of One's Own

Now we are worn out debugging other people's hardware configurations and system software deployments. We are looking to provide software-as-a-service on our hardware. We are currently mulling over the following options:

1. We charge a monthly usage fee for access to a working system that is on our premises, under our control and accessed over the wild and wooly Internet via a VPN
2. We charge a monthly usage fee for access to a working system on a host or hosts dropped by us into the data center: we set it up, keep it up and back it up: you provide power, A/C and a lack of ambient water. Seriously. From experience, we can say that the flooded machine room is a no-go.

We are cautiously excited about The Cloud; we might very well end up using Amazon's offerings in this area, once we are sure that privacy-conscious, mostly-health care clientele can dig it.

Internal IT have been very resistant so far, but we hope that accounting issues and procedural clarity with triumph. We tried to play nicely with the other children, but they kept peeing into the sandbox.

Report to Database

A common gig for my consulting company these days is plugging gaps in other vendor's infrastructures.

Once upon a time, this gig was pretty pleasant: every IT-consuming manager seemed to understand that what they bought would be, at best, 80% of what was needed, so they kept time in the schedule and money in the budget to get as much of the last 20% as they could. We were brought in early and worked with the vendor to produce a happy client.

More recently, we find that larger vendors are squeezing us out, but not by doing a great job: instead, the larger vendors are often over-promising, under-delivering, dangling the promise of cheap, expert consulting and customization that never seems to materialize. But when the initial installation is done, the deadline is past and the budget is exceeded.

As a result of this successful strategy, we are brought in when it is too late, when users are fried and management is angry, when time is short and money hard to come by.

The typical story goes like this:

1. Client considers an enterprise solution (ES) so expensive, it must do all that one could ever want. This thinking seems incredibly naive to me, but there you are.
   
2. During the sales phase, the answer to all questions is "yes, of course it does" or "of course we will" or "we are installed in X other similar companies, trust us." I would expect this to raise red flags, but it does not seem to.
   
3. During the implementation phase,  a different team from the same vendor finds many previous claims to be absurd--"I can't believe that any of our people ever told you that" is what I hear.
   
4. When the smoke clears on the installation, the client finds a given business process is not supported by the mighty ES, even by cobbling together functionality from various modules. The vendor's consulting teams come and tell the client to either stop wanting the given functionality or to wait for some bright day when that functionality is released.
   
5. For whichever of these failures simply cannot be tolerated, we are called in to plug the gap by providing the required functionality "outboard" of the ES

Given how we come into the environment, it is no surprise that our relationship with the ES vendor is not great.  Often the patch requires interaction with the ES database and support from the vendor is either terrible or non-existent. The idea that we all have the same goal (making the client happy) and the same boss (the client)  seems to be quite dead.

In this case, I often only have a database black box and a report with at least an identifier and a value for the column I seek. Even when we get schema documentation, it is usually, ahem, bare bones and, shall we say, "out of date."

So I sigh deeply, put on my favorite database hacking music and do the following:

1. I get permission to use the database (usually a secondary copy)
2. I get credentials with which to access the database
3. I am a terrifically ethical person, so I would never decompile someone else's software to get the required information. That would be wrong. Even though the client paid for the database and the software.
4. I use whatever database description hooks I have, eg MySQL's "show database" and "show tables" and "show columns" commands, or the local equivalent. I have a suite of database debugging tools leftover from a previous incarnation as a database vendor, so I can crack databases even if those databases are not relational databases
5. I write programs to take the outline of the schema and for each table:
1. get the list of columns and data types
2. select all  columns for all rows
1. check each column against the known values
2. store every hit: table:column:value
3. manually review the output, form guesses about the schema
4. write programs based on guesses, eq frequency distributions
5. refine guesses about the schema and data based on evidence

If all goes well, after a rather long time, I will have some significant clues to the schema and how the column I care about is named, in which table it lives and how it is used. I can then do what I have to do in the way of pre-processing or post-processing.

I often find out enough to continue to provide patches for various deficiencies until someone figures out that the mighty ES, which should have ended all need for IT work, is being augmented in this way and forbids any more work.

Thank God a new mighty ES seems to come along every couple of years, so I guess this cycle will keep me off the street for a while yet.

Hugs not Bugs

Ok, this week my posting is a self-pitying rant, pure and simple. The topic: asking for help from software developers.

For some reason, our industry has adopted the following value: if you need help, too bad; if you found a bug, we will do all we can for you.

At least, this is how we seem to be perceived. I spend much of many of my days at work dealing with accusations of incompetence, which accusations are actually poorly-disguised pleas for help. Whenever a client or user needs my considerable analysis and analytic skills, they ask for help by accusing me of having bugs or poor design.

Yes, I have bugs and yes, I don't have a problem with bug reports. I am highly confident that I can fix any bug and I have few enough of them that I am happy to fix whatever needs fixing. I am not talking about bug reports: I am talking about something else. Here are two examples from the recent past; one is from yesterday, in fact.


Case 1: Just Fix It!

A client reports that a system of mine which is designed to create orders is failing to create orders. She knows this because my system comes complete with an built-in audit module that told her so. My system is at the start of a very long chain, going through another department in her organization and through two other vendors' systems.

• I confirm that my system generated a request to create an order, which request was acknowledged by the next system.
• The bug report is repeated, this time with a slightly ruder, slightly more accusatory tone.
• I confirm, by looking at logs that are not my own, that the request did not reach its final destination.
• I ask the intervening department to confirm that the request went through their system and out the other side.
• I confirm that the final destination has two interfaces through which the request must pass, and that I only looked at the second of those.

I am slated to continue tracking this down, through other vendor's software as configured by other people, not because this is in any meaningful sense "my" bug but because I am good at this and am being paid to do it. They need my services, I am happy to provide those services: why does the request for service have to be framed so negatively?


Case 2: Its Not Working!

In one of my professional identities, I provide technical support for a line of technical products which I designed. This gives me contact with the retail world and a sense of how my work is actually used in the real world. Most of what I learn is terrifying and makes my consulting clients seem like ever-pleasant geniuses.

I recently fielded a "bug report" that a four-year-old version of our software had mysteriously stopped working and that if there was a patch, the customer lacked it.

This is, of course, absurd on its face. I directed tech support toward a tech blog posting I had made some time ago about debugging configuration files, since this was obviously a case of misconfiguration.

The reply, as forwarded to me, was that the data had not changed, the configuration therefore was right and that the fault lay with the software.

This was ridiculous, but arguing with someone who is either this ignorant or this belligerent is a waste of time, so instead I used our excellent configuration debugging tools, which are available free for download to any of our customers, to find and fix the small issue with her configuration: Voila! it all works--or rather, it was working all along, but now her configuration matches her data.

Naturally, it turned out that this particular user was one of the few to send a thank you note, a nice thank you note, when she got the resolution. So I try to remember that not every difficult user is a troll: some of them just need help and they apparently don't know how to ask for it.

Stone Soup

This post is about business process automation: using computers to make people more efficient and effective. These days, with computer hardware so fast and software so powerful, the most precious resource in most business settings is human attention. Computer systems should help conserve and focus human attention, not drain it away.

Growing up, I read a lot of folk tales and as an information systems consultant am often reminded of one of my favorites: Stone Soup. Here is the uninspired Wikipedia synopsis, but find a copy of the real thing if you like the idea.

The basic idea is that while a group of people are unlikely to pull together to help you, even if it helps them, they will participate in a sure thing.

I find this is a common situation with our clients: no one wants to stick their neck out to launch a project, but everyone is happy to participate in an already-going-well project, especially if participation buys them something.

Thanks the last 20% rule (apparently more correctly known as the Pareto Principle), I find that I can do the 20% of work that sets the stage on my own dime, as a proof of concept, then do the rest of the work once the client sees that success is just over the line.

In fact, once they get rolling, clients often for 120%. Why not? The first 20% was free.

So, as as customer, why not get the 80% for cheap? In theory, you shouldn't settle for the easy 80% because, as any one with experience with business process automation  can tell you, that easy 80% is what separates the bad from the adequate; that final 20% is what takes you from adequate to excellent.

I am a realist, however: for many (the majority?) of organizations, going for the easy 80% would be a big step up. So by all means, try to get it. The tricky part is knowing how much you can get out of your current environment. If the environment you have is at capacity, then you have to look elsewhere for improvement. Either get some consulting help or new software or (gasp!) put manual processes in place to make up for computer system shortfall.

Sometimes we see computer systems which are fine systems but not properly configured for their environments; in this case, if people put manual procedures in place as work-arounds, we understand. Adding manual labor is always regrettable but sometimes necessary.

But remember the story of the man, the boy and the donkey: first the boy rides and passersby chide the boy for letting his elder walk; then the man rides and the passersby chide the man for letting his child walk; finally, the man and the boy carry the donkey and the passerby mock them for doing the donkey' work. If you are carrying your donkey, you need to consider the possibility that someone isn't doing their job.

In Praise of Pausing

There are two situations in which I appear to be doing nothing: when I am thinking and when I am in what I call the Safety Pause.

Thinking

The thinking case should be obvious: as a software designer and prototype builder, a good part of my job requires thought. Thinking, at least when I do it, look like particularly quiet goofing off, with a crucial difference: I am thinking.

I accept that believing that I am thinking and not day dreaming requires a leap of faith. In order to foster trust, I try to offer what proof I can, such as white papers, design notes,rapid coding that comes from a solid plan, etc. but I find that this is often not enough to convince some people. These people seems to feel that if my fingers are not on the keyboard then I am not working. God forbid that I have coffee with a user as a part of my requirements gathering.

Don't most white collar workers have to think? Don't people take notes, write outlines, ask questions and then consider the answers to those questions? Or is there a prejudice about computer people and goofing off? Lolly gagging, frittering away time in frivolity; are these vices peculiar to computer specialists? I think not.

It is true that, as a group, we blur the lines between work and play: we play at work and we work at home sometimes. We even work in the shower sometimes (see below). I will even grant you that until I started keeping track of my time very carefully, I tended to overestimate the amount of time that I worked at home and the amount of time that I goofed off at work.




However, it is not true that I was working less that 40 hours per week and it is not true that when I need to think, I appear to be just taking up space. Coming up to me to ask what I am doing, or to jokingly imply that I am wasting time is the exact opposite of helping.

(For me, the one exception to this etiquette is the forest of unshaven young guys staring moodily into their laptops at Starbucks. I am willing to assume that they are just wrestling with writer's block or dealing with having an empty head.)

The Safety Pause

The Safety Pause is my version of the carpenter's adage "measure twice, cut once." In my consulting practice, I find it best to consider, reconsider and then pull the trigger, be the task writing code, designing software or processing data.
 
The Safety Pause lets the rest of my brain chime in and prevents "God, I just want to be done" syndrome. I embraced the Safety Pause because I kept finding that after pulling the trigger on something, the very next instant, something would occur to me that I should have done differently, or even why I should not have pulled the trigger at all. I decided that having that revelation BEFORE I pulled the trigger would be better. So I started deliberately walking away and doing something else. The thing-into-head-popping is not as quick as when I have shot myself in the foot, but it often happens.



I also find that sleeping on a problem often means that I wake up with a solution all ready for inspection. Sometimes this solution is flawed, but it is often a good starting point. I even find that ideas come to me in the shower, which is a drag because I rarely remember to bill for showering or sleeping, and yet....



Conclusion



So the next time you see a co-worker staring off into the middle distance, just leave them alone. They may actually be thinking. And if you see a co-worker finish most of a task and go get a cup of something hot before doing the final work, remember that they may just be trying to avoid trouble down the road.