Experience is Wasted on the Young

Ageism is legendary in the computer technology business: we all decry the fact that managers are pressured to hire "kids right out of college who grew up with this technology." But the pressure continues and the outcome remains the same: otherwise inexplicably bad software.

For the sake of brevity, let us choose an example of a hot new development technology: Ruby on Rails (http://rubyonrails.org/). This assumes, of course, that some thing can still be considered "hot" if I have heard of it.

If you hire a group of neophytes who "grew up with" Ruby on Rails, what do you get? You get a group of programmers who can hit the ground running in your chosen development technology, which is good. What do you lack? You lack domain knowledge in your target domain, you lack real-world project experience, you lack experience in all that a working, commercial system must do other than meet a technical spec; in fact, you lack just about all relevant experience outside the narrow domain of Ruby on Rails development.

The only scenario I can imagine being well-served by a neophyte team of Ruby on Rails programmers is a contest of some sort where the goal is to create a Ruby on Rails app in a short time that will impress a panel of college students.

Organizations keep going down this path and keep being disappointed that the software they get isn't very good. In fact, it is often worse than the software it replaces. And there isn't usually a progression within a domain, of software getting better and better, and feature sets evolving to keep the best features and lose the apparently-cool-but-actually useless ones.


This is because evolution is a cumulative process: if you keep starting from scratch, you keep making the same mistakes. This is the unavoidable truth even if you are using ever-more-hip environments to make those mistakes: perhaps you will be able to make those mistakes faster and cheaper and on more platforms than you could before, but you will still make those mistakes.

I know that there are software Mozarts out there, able to write great code from an early age, but "great code" does not mean "great software." "Great code" usually means "code that impresses other programmers." All  too often users of the software built with that code are left wondering how the author of an irritatingly hard to use program is considered so highly by his technical peers.


As a practical example from my own experience, I offer this common task: loading data into a database. I am an expert in "moving data around" so I am often called in to assist with this particular job. I often find a young, inexperienced team of programmers who are expert in a particular technology, but not computer system experts per se. Increasingly, the home team are all experts in some interactive-oriented technology, such as Ruby on Rails or Visual BASIC, and not at all comfortable with batch-oriented technology such as SQL transaction sets.

The common mistake for the young and inexperienced is to try to force a batch operation into an interactive paradigm, to write code to take the input data and force it through the system as if a superhuman typist had entered it at the keyboard. This mistake has two possible failure modes; sometimes it fails in both ways at the same time:

• The loading does not work reliably
• The loading takes an unusably long time

When the loading is unreliable, the issues are almost always exception-handling and/or data processing.

Experience tells me that usually the right way to handle a batch update is to apply the transaction one-by-one, putting aside the failures for later human review, correction and reapplication. Only rarely does the user want an entire data set rejected because of a typo in one of the fields of one of the records.

Users generally are far more comfortable with a UI to tell them how many record were in the batch, how many went applied to the database and how many were not applied. They really like being able to inspect and correct the bad records and re-process them until the batch is closed. Does this fit the textbook SQL COMMIT & ROLLBACK paradigm? No, it does not. Sadly for our team of neophytes who will have to learn the hard way why the users are so unhappy with their work. Maybe as our team gets more experience, they will get better, if they can find a job after they hit 28.

When the loading is slow, the issue is almost always trying to put a large number of transactions through a pipe designed to service human input.

The overhead in servicing human input is vast, but the human data rate is so low that this overhead is quite acceptable. Turn that equation on its head--fast data rate, no user to react to the various edit checks--and you have a very poor computing paradigm.

Instead of trying to put data through the human channel, experience tells me that it is often quick and easy and reliable to process the incoming data directly into SQL INSERT, DELETE or UPDATE statements and let the database manager do its thing. If you need to make multiple passes over the data to establish relationships, then do that. If you need to sort through transactions to weed out duplicates or just take the most recent change, do that too.

When you are done fine-tuning your batch process, you can schedule the job to run in the background and voila! yesterday's data is loaded before the start of work today. The computer doesn't mind the extra work, so make the process redundant and have it try over and over again so that delays are not a problem and so that accidental resubmission of data is rejected.

Experience tells me that these operational aspects are what makes a data loading process robust and useful. I don't see how being expert with a given development environment, but ignorant of real-world issues, would make me better at my job.

Now if I could only convince the ever-growing cadre of number-crunching, budget-watching managers who came up through the management track that this delightful fallacy is rarely a good strategy. Maybe I need to find a Ruby on Rails book and add some more buzzwords to my CV.

Report to Database

A common gig for my consulting company these days is plugging gaps in other vendor's infrastructures.

Once upon a time, this gig was pretty pleasant: every IT-consuming manager seemed to understand that what they bought would be, at best, 80% of what was needed, so they kept time in the schedule and money in the budget to get as much of the last 20% as they could. We were brought in early and worked with the vendor to produce a happy client.

More recently, we find that larger vendors are squeezing us out, but not by doing a great job: instead, the larger vendors are often over-promising, under-delivering, dangling the promise of cheap, expert consulting and customization that never seems to materialize. But when the initial installation is done, the deadline is past and the budget is exceeded.

As a result of this successful strategy, we are brought in when it is too late, when users are fried and management is angry, when time is short and money hard to come by.

The typical story goes like this:

1. Client considers an enterprise solution (ES) so expensive, it must do all that one could ever want. This thinking seems incredibly naive to me, but there you are.
   
2. During the sales phase, the answer to all questions is "yes, of course it does" or "of course we will" or "we are installed in X other similar companies, trust us." I would expect this to raise red flags, but it does not seem to.
   
3. During the implementation phase,  a different team from the same vendor finds many previous claims to be absurd--"I can't believe that any of our people ever told you that" is what I hear.
   
4. When the smoke clears on the installation, the client finds a given business process is not supported by the mighty ES, even by cobbling together functionality from various modules. The vendor's consulting teams come and tell the client to either stop wanting the given functionality or to wait for some bright day when that functionality is released.
   
5. For whichever of these failures simply cannot be tolerated, we are called in to plug the gap by providing the required functionality "outboard" of the ES

Given how we come into the environment, it is no surprise that our relationship with the ES vendor is not great.  Often the patch requires interaction with the ES database and support from the vendor is either terrible or non-existent. The idea that we all have the same goal (making the client happy) and the same boss (the client)  seems to be quite dead.

In this case, I often only have a database black box and a report with at least an identifier and a value for the column I seek. Even when we get schema documentation, it is usually, ahem, bare bones and, shall we say, "out of date."

So I sigh deeply, put on my favorite database hacking music and do the following:

1. I get permission to use the database (usually a secondary copy)
2. I get credentials with which to access the database
3. I am a terrifically ethical person, so I would never decompile someone else's software to get the required information. That would be wrong. Even though the client paid for the database and the software.
4. I use whatever database description hooks I have, eg MySQL's "show database" and "show tables" and "show columns" commands, or the local equivalent. I have a suite of database debugging tools leftover from a previous incarnation as a database vendor, so I can crack databases even if those databases are not relational databases
5. I write programs to take the outline of the schema and for each table:
1. get the list of columns and data types
2. select all  columns for all rows
1. check each column against the known values
2. store every hit: table:column:value
3. manually review the output, form guesses about the schema
4. write programs based on guesses, eq frequency distributions
5. refine guesses about the schema and data based on evidence

If all goes well, after a rather long time, I will have some significant clues to the schema and how the column I care about is named, in which table it lives and how it is used. I can then do what I have to do in the way of pre-processing or post-processing.

I often find out enough to continue to provide patches for various deficiencies until someone figures out that the mighty ES, which should have ended all need for IT work, is being augmented in this way and forbids any more work.

Thank God a new mighty ES seems to come along every couple of years, so I guess this cycle will keep me off the street for a while yet.