The Utility of Bad Examples

Today I am reflecting on lessons I have learned in my long career and how I learned them. The obvious approach is to remember the exceptional individuals who were great at their jobs while being pleasant and consistent and generally a pleasure to know. But who needs to be reminded that we can learn from paragons of excellence? Instead, I am contemplating what I learned from the deeply incompetent, the arrogant and the people I've met who cannot seem to learn from their negative experiences.

Many of us really love a good story about some fathead and his or her hideous blunders but that is what I present here. Like the poor, fatheads will always be with us. Avoiding them is the best course of action, if you have that luxury. Failing that, there are strategies for mitigating the ill effects of fatheadedness, but that is entirely different story. Besides, we often have no choice but let fatheadedness run its course. In those all-too-frequent cases, what is the benefit?

The benefit is experience, precious useful experience and the attendant opportunity to learn from someone else's mistakes. In post, I contemplate the utility of bad examples: what do the actions of fatheads teach us, to make us better at what we do?

In my experience, the truly spectacular fathead (SFH) is arrogant: so arrogant that he or she (usually a he, so I will use that pronoun for convenience) is above mere conventional wisdom. Sometimes "thinking outside the box" is valuable, but most of the time thinking outside the box is a waste of time or worse: for conventional situations, conventional wisdom usually suffices. In order to be worth it, thinking outside the box must present some clear additional benefit to compensate for the greater effort. I try to keep in mind that in today's IT environment, human attention is the most precious resource. Use it wisely.

(So why the endless praise of out-of-the-box thinking? Because it is the employment situation that gives kudos for doing the obvious, even if the obvious is the right choice. And, every once in a great while, thinking outside of the box saves the day. The trick is to pay attention so that you notice when your situation is abnormal, but that is another post.)

Since the SFH is arrogant, he is often an object lesson in understanding conventional wisdom: why do we never do that? Oh, THAT's why. Rather than recount amusing examples of stupid people doing stupid things, I offer this suggestion: keep track of instances of bold, innovative thinking. Follow up months or years later: did the bold innovation do better or worse that conventional wisdom expects? Why? I have learned much about why certain rules of thumb exist this way--mostly as I sat amid the smoking wreckage of some IT disaster, but at least I had something useful to do while I was sitting there.

Often the SFH makes the same kinds of mistake over and over again; after all, it is almost a requirement that the SFH be unable to learn from his mistakes if he is to remain the SFH. Once you see that patterns, the SFH is useful as the embodiment of a certain kind of habitual error: when faces with certain kinds of decision, ask yourself what the SFH would do and then DO SOMETHING ELSE. Ideally, do something that makes sense in light of what your SFH has inadvertently taught you.

Now we come to the most painful methodology: seeking useful feedback. I find that a simple description of an SFH strategy, preferably to someone outside the SFH's organization to avoid accidental embarrassment, often nets useful results. A very good outcome of bouncing your observations off of someone you respect is that they may explain levels to the problem you had not seen. The most useful, but least pleasant outcome of this exercise is the casual observation from your respected sounding board that YOU are guilty of the same bad judgment. If true, this feedback is invaluable in improving yourself.

I say "if true" because some people feel that it is either appropriate or polite to accuse the speaker of any fault the speaker finds in others. But this is not a get-out-of-jail card: not all unpleasant feedback is some kind of tit-for-tat reflex. This means that really have to think about that you said, what they responded and how their insight improves your understanding of your situation. If your understanding is not improved, then discard the feedback--but discard it discreetly and politely. You might need feedback in the future and remember: most people won't be willing to give it at all, so sometimes reflexly negative feedback is better than no feedback at all.

Ignorance is bliss--until it isn't. Think about why bad decisions turned out to be bad and you are on the path to making better decisions in the future.

Reasoning From Worst Principles

I have nothing against reasoning from first principles; in fact, I really enjoy it. Reasoning from first principles is an excellent alternative when experience and expertise fail us. I particularly like reasoning from first principles when helping people debug large, complex systems: I find that going back to basics very helpful in avoiding the prejudices and preconceptions that are usually at the heart of really thorny debugging sagas.

However, there is a situation in which I find reasoning from first principles to be very frustrating: when management values their reasoning from first principles over all other input, including experience and expertise.

I understand that modern IT decisions are complex and cover such a vast array of possibilities that making a truly informed decision is often impossible. But we should not let the fact that we can't be perfect provide us with an excuse to be terrible.

I find that in the face of uncertainty only flexibility offers a high probability of success.

To a recent example from my own consultancy, we are in the midst of adding disk space to our core server. This is a very common situation, although there are a few complicating factors:

• Our server is a High Availability Linux cluster with a shared RAID
• Our server is a bit long in the tooth
• We really prize our server uptime

 In consultation with our sys admin experts, I was presented with the following options:

1. Get a new server, which would be more current and have more capacity
2. In-place hard disk upgrades
3. External disk upgrades

I evaluated the options thusly:

1.  New server
1. Pros: effective, easy fall-back, modernized environment
2. Cons: expensive, housing old & new, possible environment issues, not indicated except for disk space
2. In-place upgrade
1. Pros: leaves most of server intact
2. Cons: always a risk to crack open a case, requires lots of human attention, process is complex
3.  External disk upgrades
1. Pros: cheap in hardware cost and human time, minimum disruption to servers
2. Cons: probably mediocre performance, another point of failure, chance to trip over cable, etc

I chose option 3 as being the cheapest, fastest, safest option.

Given what happened next, I was probably wrong:

• the first external SATA enclosures did not play well with the kernel
• the second external SATA + disk combination takes too long to wake up
• in rearranging the enclosures showed up an issue with the outdated internal disk on the primary node

So now it looks like we will have to do option 3 for now and look into option 2 as well. Oh, well: that's life when you don't know everything. At least the server has stayed up and useful, we haven't more than we can afford and I see a path to my goal. I had to change course with the first set of enclosures and now I will have to change course again. Responding to real-world consequences of actions is how adults deal with reality.

Sadly, in our information systems consulting, we are running into magical thinking with ever-greater frequency. Specifically, we are running into the following template:

We had a problem, P, which we needed to solve. To solve P, we had to decide on a solution. That decision is decision D. In order to make decision D, we over-simplified P and declared that system S would completely solve P. We paid so much for S that we reason, from first principles, that problem P is solved. Since this is the real world and therefore complicated, there are aspects of system S which are imperfect, but we declare that since system S cost so much, it must do so much and it must do enough.

This is not simply a case of my being nit-picky and snarky about the usual inability to think clearly. In this case, the managers have defined the problem as not existing. Fixing a problem that does not exist is either impossible or unnecessary  or both.

When this magical thinking is in place, the tension that our customers live under is huge and is caused by the fact that they live with problems but are declared to be without problems.

There is usually the added tension of the implication that not being able to use the expensive solution to solve their problems makes them incompetent; there is often the complication that someone in their chain of command had to consciously over-promise in order to get the money for the expensive solution, so trying to get help is viewed as disloyal.

As an outsider trying to solve the problem, it is very uncomfortable to be told that whatever else we do, we may not name the actual problem since the actual problem is officially nonexistent.

This gives us, the outside consultants, two options: either look for related solutions which might help the actual problem, or be heretical and name the actual problem. Sadly for us, many professionals view consultants as a kind of Foreign Legion: expendable troops. If we poison our relationship with their bosses by saying what they cannot, so what?

We understand that only we care about how uncomfortable or unpleasant our working conditions are and there is good money to be made as IT Foreign Legionnaires, so we cannot really complain about that. But what I can do is point out that once you enter the realm of magical thinking, of reasoning from worst principles, you make success almost impossible and you make your own working environment tense and unproductive. Who needs that?

Benign Neglect & Tech Support

In general, I am not a fan of the "benign neglect" concept, but I make an exception for some tech support questions.

(Psst, I'll tell you a secret if you promise to keep it to yourself: I am not the only one. But more on that below.)

By this, I mean that when I am providing tech support for products, which I do on occasion, I sometimes deliberately take my time responding. I sometimes take a long time because I am really busy, but that is an entirely different kettle of fish.

I should point out that my software company, as opposed to my consulting company, is a no-frills operation: we offer a high-quality product at a rock-bottom price. We do this by skimping on the human interaction: we provide a web site, a tech blog and a free on-line manual. What we don't provide is a call center filled with bodies to talk to you on the phone; we also provide email-based tech support and a cheerful, prompt, full refund if you would rather pay literally ten times as much to one of our competitors, who will happily talk to you on the phone.

So we have a severe market-segmentation device; them that like us love us and them that don't like us REALLY don't like us. Those that love us love the fact that when we get back to you, you get a thorough, thoughtful answer. But before you get that thorough, thoughtful answer you may get a thorough, thoughtful silence.

And not all tech support requests trigger the benign neglect policy: in fact, most do not. Most are reasonable questions from reasonable people which are a pleasure to answer. Some are valid criticisms that I cringe to acknowledge, but look forward to fixing. Then there are...the others.

There are two query categories to which I apply this policy: the breathless and the clueless.

By "breathless" I mean the really excited highly caffeinated poorly punctuated query that runs on and on and interrupts itself because the writer is so darn excited he-or-she cannot contain his-or-herself and just goes on and on with super-detailed but indiscriminatingly detailed accounts of some tale of horror involving my technology behaving in ways that it does not cannot never does behave.

By clueless, I mean the complaint or question reveals such a profound ignorance of the general topic and specific issue that it is hard to know where to begin to answer.

I give the breathless a chance to catch their breath, which they often do: the initial babbling request for tech support is often followed by the equivalent of "oops, sorry, I figured it out, I was doing something stupid." In other cases, I at least get a follow up that says "I figured out X and Y, but I still don't get Z." That is a query to which I can respond: I feel that we have a shared basic understanding and a clear context for a reply to issue Z, instead of a headache and desire to forget.

I give the clueless a chance to get a clue: to reconsider what they are trying to do or how they are trying to do it. Or to decide that they want one of those cheerful, prompt and full refunds. Which we are delighted to provide: spending time arguing with people about how a $250 product can't come with hours of hand-holding is a great way to lose money.

PS It is my experience that every tech support organization does something similar: they may have a polite ignoramus contact you to acknowledge your request for support, but the actual tech firepower is probably waiting for you to catch your breath or get a clue.

Experience is Wasted on the Young

Ageism is legendary in the computer technology business: we all decry the fact that managers are pressured to hire "kids right out of college who grew up with this technology." But the pressure continues and the outcome remains the same: otherwise inexplicably bad software.

For the sake of brevity, let us choose an example of a hot new development technology: Ruby on Rails (http://rubyonrails.org/). This assumes, of course, that some thing can still be considered "hot" if I have heard of it.

If you hire a group of neophytes who "grew up with" Ruby on Rails, what do you get? You get a group of programmers who can hit the ground running in your chosen development technology, which is good. What do you lack? You lack domain knowledge in your target domain, you lack real-world project experience, you lack experience in all that a working, commercial system must do other than meet a technical spec; in fact, you lack just about all relevant experience outside the narrow domain of Ruby on Rails development.

The only scenario I can imagine being well-served by a neophyte team of Ruby on Rails programmers is a contest of some sort where the goal is to create a Ruby on Rails app in a short time that will impress a panel of college students.

Organizations keep going down this path and keep being disappointed that the software they get isn't very good. In fact, it is often worse than the software it replaces. And there isn't usually a progression within a domain, of software getting better and better, and feature sets evolving to keep the best features and lose the apparently-cool-but-actually useless ones.


This is because evolution is a cumulative process: if you keep starting from scratch, you keep making the same mistakes. This is the unavoidable truth even if you are using ever-more-hip environments to make those mistakes: perhaps you will be able to make those mistakes faster and cheaper and on more platforms than you could before, but you will still make those mistakes.

I know that there are software Mozarts out there, able to write great code from an early age, but "great code" does not mean "great software." "Great code" usually means "code that impresses other programmers." All  too often users of the software built with that code are left wondering how the author of an irritatingly hard to use program is considered so highly by his technical peers.


As a practical example from my own experience, I offer this common task: loading data into a database. I am an expert in "moving data around" so I am often called in to assist with this particular job. I often find a young, inexperienced team of programmers who are expert in a particular technology, but not computer system experts per se. Increasingly, the home team are all experts in some interactive-oriented technology, such as Ruby on Rails or Visual BASIC, and not at all comfortable with batch-oriented technology such as SQL transaction sets.

The common mistake for the young and inexperienced is to try to force a batch operation into an interactive paradigm, to write code to take the input data and force it through the system as if a superhuman typist had entered it at the keyboard. This mistake has two possible failure modes; sometimes it fails in both ways at the same time:

• The loading does not work reliably
• The loading takes an unusably long time

When the loading is unreliable, the issues are almost always exception-handling and/or data processing.

Experience tells me that usually the right way to handle a batch update is to apply the transaction one-by-one, putting aside the failures for later human review, correction and reapplication. Only rarely does the user want an entire data set rejected because of a typo in one of the fields of one of the records.

Users generally are far more comfortable with a UI to tell them how many record were in the batch, how many went applied to the database and how many were not applied. They really like being able to inspect and correct the bad records and re-process them until the batch is closed. Does this fit the textbook SQL COMMIT & ROLLBACK paradigm? No, it does not. Sadly for our team of neophytes who will have to learn the hard way why the users are so unhappy with their work. Maybe as our team gets more experience, they will get better, if they can find a job after they hit 28.

When the loading is slow, the issue is almost always trying to put a large number of transactions through a pipe designed to service human input.

The overhead in servicing human input is vast, but the human data rate is so low that this overhead is quite acceptable. Turn that equation on its head--fast data rate, no user to react to the various edit checks--and you have a very poor computing paradigm.

Instead of trying to put data through the human channel, experience tells me that it is often quick and easy and reliable to process the incoming data directly into SQL INSERT, DELETE or UPDATE statements and let the database manager do its thing. If you need to make multiple passes over the data to establish relationships, then do that. If you need to sort through transactions to weed out duplicates or just take the most recent change, do that too.

When you are done fine-tuning your batch process, you can schedule the job to run in the background and voila! yesterday's data is loaded before the start of work today. The computer doesn't mind the extra work, so make the process redundant and have it try over and over again so that delays are not a problem and so that accidental resubmission of data is rejected.

Experience tells me that these operational aspects are what makes a data loading process robust and useful. I don't see how being expert with a given development environment, but ignorant of real-world issues, would make me better at my job.

Now if I could only convince the ever-growing cadre of number-crunching, budget-watching managers who came up through the management track that this delightful fallacy is rarely a good strategy. Maybe I need to find a Ruby on Rails book and add some more buzzwords to my CV.

Little Things Mean Alot, Eventually

Being a science-oriented male, I have a perhaps under nuanced view of the world. At both work and home, I tend to prioritize my tasks, which is usually good.

But when there is not enough time to do everything, this method has a huge drawback: because there is always a high priority task to do, low priority tasks never get done. This causes problems both at work and home: eventually the owners and stakeholders of even low priority tasks tire of waiting for attention.

As is so often the case, I drew on Linux kernel development for inspiration in my life. [This is a huge exaggeration for comic effect. Really. I hope.]

In the early days, the Linux kernel's scheduling was relatively simple: the more important tasks, such as disk I/O, had high priority and relatively un-time-critical tasks such as servicing the keyboard had relatively low priority. This meant that early Linux systems were amazingly good at getting performance out of mediocre hardware BUT performance degradation was not graceful. In other words, when things went bad they went very bad very quickly.

I remember hammering on the keyboard in frustration, having provoked a database server into taking on more work than it could handle, my typing apparently having no effect, until the system decided to see what was happening in keyboard land, whereupon all of my frustrated input, all of it, was faithfully processed to amusingly ill effect. Well, amusing with the distance of 19 years. At the time, I was beside myself.

Soon thereafter, the raise-the-priority-with-age algorithms were fine-tuned, giving me a much better experience even when I do something really stupid. Or when other people do something stupid. When stupidity raises its ugly head.

Human beings seem to have an innate sense that age should raise the priority of a task. Intellectually, we all want to do the most important tasks (or easiest, or most praise-garnering tasks) but viscerally we also want the small things which have been bugging us forever to get done.

As IT people, we often forget that. I know that I am inclined to avoid what I consider "trivia" because I feel that I am so important that such things are a waste of my time. Alas, I cannot seem to get much buy-in to this view of the importance of my time, so I  had to institute "laundry day" in our consultancy: a day every couple of weeks dedicated to doing the laundry: fixing or adding all the small, silly, useless things that users value and that they will eventually rage about unless we get to them.

We see many IT organizations which do not make time for trivia, to very ill effect. We sympathize with IT folks who are not given this opportunity: if your boss does not let you do the laundry, it isn't your fault that you stink (figuratively speaking: the tendency of programmers toward low standards of hygene is another matter entirely).

Which reminds me: currently, I am WAY overdue to replace one of the outside lights, which lights are important to my wife but not to me. I will try to get to that today even if I feel that I have more important things to do. Sigh.

Uneven Distribution of IT Talent or Expectation?

Once upon a time, I wrote a book about IT management. This was something of a vanity project: I got some stuff off my chest and then I felt better. (This was before blogs, which are much better suited for venting.)

Since I have a side business publishing books, this vanity project at least had some practical value: we debugged the process with my vanity project before moving on to other projects.

However it came to be, I was struck by the reaction to my book: the few people who read it fall neatly into one of two camps:

1. This book is useless: it describes solutions to management problems that no one has because no IT environment is this bad.
   
2. This book is a much-needed guide to the horror that is current American corporate IT.

This bifurcated reaction raises a question, regardless of my book per se: is IT talent massively unevenly distributed, or does expectation or understanding of IT performance vary massively from organization to organization?

I have not been able to make much headway answering this question: many of the people I have surveyed are MBAs who seem both confident in their ability to assess IT and greatly lacking in actual ability to assess IT at a fundamental level.

On the other hand, since IT management is increasingly about pleasing MBAs,  perhaps I just out of date about what constitutes IT competence.

In my consulting practice, we see a wide range of IT competence with a strong bias toward low competence. So I am inclined to believe that talent is vastly unevenly distributed. But why would that be? Is financial and accounting talent similarly unevenly distributed?

Many colleagues have pointed out to me that this bias in my observation is to be expected: we are consulting firm, after all, and only incompetent organizations need our services.

I am not a big fan of this idea: I maintain that rational, competent IT organizations can choose to outsource specific jobs for any or all of the following reasons:

1. There is not enough time to acquire the expertise in-house.
2. This is a pilot project with an uncertain future.
3. You are not a development shop and cannot find an off-the-shelf solution.

In the past, we have done project with companies internationally recognized for their excellence, so I dispute the notion that only weak organizations consume IT consulting, even though there is some of that.

My quest continues: someday I may know if the talent or the expectation is what varies so greatly from organization to organization.

Almost Never Say Die

In designing, creating, deploying and supporting information systems, I follow the military model: I call lower-level decisions and tasks "tactical" and higher-level ones "strategic."

Like many (most? all?) successful people, I hate to fail and I don't do it very often, at least at the strategic level. In fact, I can't remember ever truly failing at the strategic level. Failure at the strategic level is usually catastrophic: entire projects which are never used, technology that just doesn't work, solutions that either do not solve the problem or solve the wrong problem. Avoiding strategic failure is usually matter of either knowing your problem space or recognizing early that your solution is not right.

In a sense, I fail at the tactical level all the time in the sense that I try out approaches and reject them if they do not work sufficiently well. I very rarely, if ever, simply throw up my hands in despair and tell the users to live with it.

(I do sometimes tell the users that their boss won't pay for the fix, but that is a different story, even if that story looks the same to the poor users.)

The key to useful failure, as opposed to disastrous failure, is a sense of what should be. How do you know that this job is taking too long and that a different approach is warranted? Because you have a sense of how long the job should take.

How do you know when a restructuring, an expensive, time-consuming, bug-introducing restructuring is required in order to support an apparently trivial upgrade? Because you have a sense of the present time and energy the lack of restructuring is causing and the future pain that the restructuring will avoid.

For example, I threw in the towel yesterday: I was supposed to complete a relatively minor upgrade to one of our apps. The implementation was complicated by the fact that the client had insisted that databases were too big a hammer to use to crack the walnut of configuration, so this app uses text files to store records and support a simple Web-base editor UI. I realized that having extended and re-jiggered this code for years, this final upgrade was a bridge too far: a job that should have taken an hour or two was going to take much longer. In fact, after an hour, I had just about figured out my approach.

I was also behind on delivery because I kept avoiding this project because I knew, deep down, that it would take too long and have too many bugs in the initial implementation and that it would, by my standards, be a failure. And I hate to fail.

Yesterday, after my planning and before my coding, I had to admit to myself that the right thing to do is to rewrite the entire configuration handling, basing it on...a database, as one would expect. This will make the editor easy--and easily stolen, eh re-used, from other parts of the same project. This will make the current and future changes easier, faster and safer. This will make the end user experience better than it is now. This will make the apps which use the configuration information simpler and more dependable. It is the right thing to do.

I will still fail to deliver this upgrade in what I consider to be a timely manner. But I will succeed in the upgrade and I will make the app better, faster and more robust.

One of the joys of getting older is the ability to recognize tactical failure, or impending tactical failure because once you recognize it, you can avoid it or side-step it.

Here I swerve into the uncomfortable area of ageism: the flip side of our industry's love affair with youth and energy over age and experience is a frequent lack of strategic sense. I cannot count the number of times I have had to tell a young programmer, who has proudly shown me his implementation of something, "it took you 12 hours to do a 4 hour job: you should have come to me for help instead of plowing ahead." I don't care that he (always he) stayed at his desk for 12 hours. I don't care that he did it "himself." I don't care that he feels that his implementation is special; alas, I only care that time and energy was wasted.

I am middle-aged: I no longer am willing to code for 14 hours at my desk, bullishly pounding away until the tactical job at hand is done. I haven't tried in years; perhaps I simply cannot do it any more. I hope never find out, because I am now sure that only strategic failure requires such tactical sacrifice.

On the other hand, I also no longer end up having people, when I am done with my death march, asking me "why did you spend all that time on that job? Couldn't you sense that something was wrong?"

I can sense that something is wrong. I can even admit it. And that makes all the difference.