Horse Trading, Deadlines & Budgets

Here is another take on my philosophy that the classical software engineering model is a poor fit for business process automation. This week's topic: horse trading, ie "you give me something and I'll give you something else."

As I understand it, this is the classical software engineering model:

1. Encounter a problem which can be solved or ameliorated with IT
2. Do extensive requirements gathering
3. Define a clear and fixed scope-of-work
4. Write a detailed design of an implementation which meets the requirements
5. Create an implementation plan with dates and milestones
6. Develop, debug, deploy

This is fine for many situations, especially situations which require the creation of something new that is complex and a team effort. I have participated in projects using this model in nearly every role and at nearly every level.

But I found this that model is poorly suited to business process automation, partly because this model is good at building something from scratch but business process automation is almost always a renovation, not a new construction.

A key way in which this model does not suit business process automation is this model's lack of accounting for horse trading, for the give-and-take required to deploy a solution in a working business.

(Horse trading also makes the apparent scope merely the tip of the iceberg, but we will cover that indirectly, below.)

Rather than define horse trading in the abstract, I will take the easy way out and give an example.

My firm specializes in medical information systems, particularly interfaces between medical information systems. Our client had an unusual situation: in addition to their Main Lab, they have a collection of loosely affiliated labs. To the outside world, this distinction is confusing and uninteresting, so this situation created a PR problem: why were some orders slower to process and harder to get results from?

The answer was the invisible divisions between the Main Lab and the Other Labs, but no one really wanted to know why, they just wanted better service. Once the customer grumpiness reached critical levels, the project became a crisis. Since we specialize in the difficult and the critical we were called in because decades of previous attempts had failed.

We were able to connect the three most important Other Labs, one at a time, quickly and cheaply. The secret? Horse trading and the flexibility horse trading requires.

In theory, the job is easy:

1. encode Other Lab results in an industry-standard format (HL7)
2. create a simple TCP/IP connection between us and Main Lab
3. send message in HL7 over TCP/IP connection

Our competitors bid about $50K per connection. We could do this part for $17K for all connections and still have made money, but we knew that the job was not this simple, or it would have already been done. We bid a base of $17K and about the same for each connection. We made money and got the job done, but here is how the project unfolded:

Lab A only processed orders from the Main Lab's parent organization and they had no internal computer system, so they wanted a simple work flow support app in exchange for going along with the computerization. So in addition to the straight interface, we created app A, which is driven entirely from Main Lab order for this lab and which lets Lab A see upcoming orders, confirm receipt of specimens, enter results and create local reports in case their customers want paper.

Lab B processed mostly orders from the Main Lab's parent organization and they also had no internal computer system, but they had automated analysers from which they wanted results automatically entered. So in addition to the straight interface, we created app B which is driven off of the analyser output, has different reports and pulls patient demographics from the Main Lab to flesh out the reports.

Lab C processed mostly orders from outside the Main Lab's parent organization. As a research-oriented organization, they had their own database of results kept in a homegrown app. They wanted to subcontract for a better internal app and database, which we did. We also created app C, a much simpler app mostly about importing results from their new internal app, allowing for verification and electronic signing of the results which were then passed along to the straight interface.

The project ended up giving everyone what they needed and being cheaper to boot. But  the way in which the project played out does not fit the classical model at all.

Would a classical requirements-gathering process have given us the true scope beforehand? I doubt it, since the outside players have no incentive to reveal their agenda until they are sure that they will get what they want. Often players in business processes do not know what they want until the new situation becomes clearer, nor do they know what is actually possible until they seem some progress.

So remember: not all project scope changes are caused unethical consultants or incompetent engineers. Bear in mind that business process automation is almost always renovation and that renovation is almost always a compromise. Renovation often means working around outdated decisions and respecting the need for on-going operations.

Some jobs are bigger than you can see and some situations need a little good faith horse trading in order to arrive at mutually beneficial solution.

Getting There...Eventually

Something happened recently in my consulting practice that used to drive me crazy. But thanks to experience, ennui, or both, I don't object any more.

The issue was with an interaction with a client. The situation is all-too familiar: I am creating a tool for the client and he wants to good fast & cheap, which is what client seem to think "agile" means.

As part of this speed and lower cost, he keeps choosing the minimal solution to every problem. He isn't really interested in domain expertise or 30 years experience creating these kind of apps: he knows what he needs and he knows what he wants.

Specifically, he knew that he didn't need help choosing catalogue items on which to report, even though I have had to create multiple look-up tools in the past because choosing the exact catalogue item you mean is complex and difficult. Part of the complication is that the data set contains orders from the previous ordering system, which means that one needs to know the appropriate ordering system, based on order date.

However, he knew that he did not want any bells or whistles, so version 0 was a hypertext link to the legacy web version of their catalogue. After all, he and his colleagues are familiar with the current catalogue, thank you very much.

He was able to find at least some legacy catalogue numbers, but with difficulty. Turns out he didn't know the current catalogue as well as he thought: version 1 added a link to the current web version of their catalogue.

As beta-testing went on, he complained about how hard it was to find the right items, especially if one wanted to report on dates which span the two ordering systems. I showed him one of my interactive look-up tools and he found it usable, but wanting. So version 2 is adding a revamped version of my tool the prototype.

Will we get to where I started, a heavily expert interactive look-up tool? Yes, I think that we will. Is this frustrating? No, not this time around: the client is happy because he is driving the development and is confident that he is not getting more programming than he needs or wants. I am happy because this is relatively untaxing work in a down economy; in fact, I suspect that this indirect route will end up paying me more than letting me do my thing would.

Younger me would have chafed at the lack of respect and taking of credit for my ideas and my work. Current me say, relax, keep your eyes on the prize and get your affirmation somewhere else: IT consulting is unlikely to be a good source of affirmation, at least in my experience.

Why Is Interfacing So Hard?

In theory, we provide high-end, sophisticated data analysis and reporting tools. In practice, we provide system-to-system interfaces with data analysis attached. This is because system-to-system interfacing is really hard and is usually done really badly.

We are often asked "why is this so hard?" (sometimes asked as "why were you able to do that when our people could not?" and "why did that take so long?"). So this post is all about why interfacing is hard.

Conceptually, interfacing is easy: one system ("the source") exports a datagram (a complete unit of information) and the other system ("the destination") imports the datagram. In theory, there are many "standard" formats for electronic data interchange if both systems support at least one common format, all should be well.

But all is rarely well in the interface domain. Why not?

The most common source of pain is the mismatch of data models. This sounds complicated, but it isn't and is easily explained with an example:

Consider two medical information systems which we will call A and B. System A is patient-based: the patient is the primary index and activity hangs off patients. System B, on the other hand, is visit-based: the visit is the primary index and activity hangs off visits.

System A: patient->visit->activity

System B: visit->patient->activity

The patient-centered data model is better for patient-centered analysis: it is easy to get all the data associated with a given patient for a particular time period. But the visit-centered model is better for keeping track of the current state of the system.

Regardless of which model is "better" the interfacing is complicated by the fact that the frame of reference is different. Let us work through an example.

1. System A exports a datagram which contains a visit. System A is patient-based, so the visit is exported as a small part of the patient's data.
2. System B imports the datagram
1. translates the datagram into its native System B format
2. inserts the contents of the datagram into its database

The tricky part is how to resolve ambiguity: System B is visit-based, so if there are issues with the patient demographics, who cares? We can insert the visit and worry about the patient later. But if there are ambiguities with the visit, that is fatal to System B. However, System A has the opposite worldview: so long as the patient is properly identified, the visit can ride along. So System A may have bogus visits (duplicates, unmerged visits, etc) which will choke System B, while System B may have bogus patients (duplicates, patients identified as both Bill and William, etc) which will choke System A.

When the interfacing goes awry, both vendors will be huffy: our system is fine, our export-or-import is correct, the problem must be on the other end. And they are both sort of right.

In our experience, this is where interfacing often goes off the rails: when a mismatch between the two systems would require one or other end to bend. Not only do most vendors defend their model to the death, but many vendors can only imagine their own model: any other model is "wrong". The customer gets caught in the middle.

This is where we come in. If all goes smoothly, an interface is no more than a way to convey an export from one system to be imported into another system. However, in the real world, we have often had to build intermediate systems which take in data with one model, transform that data into another model and export from the transformed model.

So if your systems "don't talk to each other" data model mismatch is a good bet as the culprit. And if your internal group or consultant or vendor quotes a larger-than-expected cost to fix the issue, there is a chance that you are not being ripped off; instead, if you are luckly, you are being forced to pay to bridge the models.

Paperless, ha! or, Your Seams are Showing

One of our clients recently had to back off yet another paperless initiative. We are often asked about this very thing: why is it so hard to go paperless, even here in the early days of the 21st century? We thought that this particular instance is a good illustration of a general problem.






In this case, there are many computer systems involved. There is either no integration between these systems or rather modest integration. Here is the process in question:

1. The customer either uses a third-party app to create an order, which is printed out, or writes directly on a piece of a paper, or uses the preferred app which creates an order, transmits it electronically and prints out the paper.
   
   
   
2. The paper order is brought to an intake center. In theory, the paper is just a security blanket and the order ID brings up an electronic version of the order. The intake is done, which creates an order in the system used to process the order.
   
3. The results of intake and the electronic order arrive in the processing system. The order is processed. The resulting status is sent yet another system, the one which supports status lookup.

Sounds a bit more complicated than one would like. There are more and different systems than one might expect. But historical accident and practical considerations make the process a bit more complex than theory would dictate.

In theory, the integration across systems should be real-time and seamless, which would make the number of actual systems in play almost irrelevant, so long as they all work together. Poor theory, it never gets its way in the IT world.

So why does the paper still exists? Well, consider a real-life scenario from the very recent past:

• The customer creates an order for items I1, I2, I3, I4, I5
• The intake system does not understand I5, leaving I1, I2, I3 & I4
• The processing system does not understand I4, leaving I1, I2 & I3
• The status system proudly displays an order for I1, I2 & I3
• Irate customer calls demanding to know what happened to I4 & I5
• Baffled customer support has no idea those items ever were ordered
• The customer is very dissatisfied
• We are called in to help investigate and find item rejections in logs
• The client sees how this is happening, re-instates the paper order

So the paper copy of the order is once again riding along with the rest of the material. Now when the same thing happens, but the process can compensate.
Instead of an entry in an interface log somewhere, the intake clerk can see that item I5 is an old code, for which there is a new code and the receiving clerk can see that I4 conflicts with I3 and that the customer must have meant to order something else entirely.

In this case, the paper allows the original intent to be accessible at every stage to all the interested parties, in a form everyone can read. If the integration of the various systems was perfect, was "seamless," then there would be no job for the paper to do.

Alas, the integration is imperfect and the seams are very much in evidence. So the paper stays, for now. But pity the poor systems implementer who is supposed to fix this problem, because that job is very hard. Electronic orders arrive in the background; there is no User Interface for systems interfaces and there is no user looking when the order is being processed. So how best to flag orders with issues? Whose responsibility is it to fix the issues?

In our practice, we find that logging is good, flagging is appreciated, but documentation is king: we put the error message as comment on the order so that, at every stage, everyone can see what happened. Specifically, the users who understand the process can see the messages, instead of sys admins who rarely can do anything about business process issue.

We suspect that so long as system integration has seams, critical processes will either have paper or regular failure. So don't sell that paper company stock just yet.

In Praise of a Touch of Humility

I am not a humble person. I don't think that I ever have been. So at first blush it may seem strange that I would be praising humility. In fact, I claim that this is a natural situation: I pride myself on being effective and accurate. In order to be effective and accurate, I need good information and feedback. In order to get good information and feedback, I need to seek it, which requires embracing the fact that I make mistakes and therefore trying to avoid future mistakes and fix past mistakes.

I hasten to point out that I do not think that I suffer from the over-confidence that infects so many leaders in so many workplaces. (I have already ranted about the childish pretence of endless certainty in a previous post but this is a new, slightly different rant.)

As a practical matter, this means that I approach software development in a specific way: I do things more than once. I write "watch dog" programs to monitor processes. I write and use consistency checks. On the plus side, many of my mistakes are caught. On the minus side, many of my mistakes are exposed to public view. However, I sleep better at night knowing that my confidence in my work is based on more than my high self-regard.

As a theoretical matter, consider this interesting Harvard Business Review blog entry: Less-Confident People Are More Successful by Tomas Chamorro-Premuzic. His thesis is this:

There is no bigger cliché in business psychology than the idea that high self-confidence is key to career success. It is time to debunk this myth. In fact, low self-confidence is more likely to make you successful.

After many years of researching and consulting on talent, I've come to the conclusion that self-confidence is only helpful when it's low. Sure, extremely low confidence is not helpful: it inhibits performance by inducing fear, worry, and stress, which may drive people to give up sooner or later. But just-low-enough confidence can help you recalibrate your goals so they are (a) more realistic and (b) attainable. Is that really a problem? Not everyone can be CEO of Coca Cola or the next Steve Jobs.

As a side note,  I point out that a friend of mine offered a gloss on this article for me: he would say that we are talking about humility here, not clinically low self-esteem. I would define humility as the trait of being willing to consider negative feedback; I suspect that there are many working definitions. The dictionary definition I found at www.dictionary.com is this:


 modest opinion or estimate of one's own importance, rank, etc.


I would say "unexaggerated" is a key component of this concept, but perhaps that is only my personal sense.

Regardless of the fine print on the definition of the opposite of over-confidence, I fervently hope Chamorro-Premuzic is right in his assertion that not-overly-confident people are more successful. In my bitter experience, boundless and baseless confidence are richly rewarded, without regard to the outcomes of those manly, clear-cut and confident decisions when complex, nuanced, multi-stage decisions would seem to be required. (With lots of checking to make sure that the path along which we are all running is the right path.)

It would be fabulous to put the Era of Empty Assertion behind us and move on to a more results-based, fact-based, reality-based, merit-based workplace, at least in IT. May it be so.

IT Ecosystems or Many Small Benefits

I find my smart phone to be a terrific tool and a significant advance over what went before. I have been contemplating this recently because I see a similarity between trying to explain *why* I am happy with my smart phone to my father-in-law and trying to explain why systemic IT improvements make sense to some of my clients.

In both cases, the audience does not share my basic premises. In both cases, there is no "killer app" and no clear analogy from what they know to what they do not know. In both cases, I am asking for a leap of faith.

For my aged father-in-law, who does not particularly want people to be able to interrupt his day with their inane chatter, the mobile phone concept never made sense. Now, when I try to explain that telephony is the minority of what I want my smart phone to do, he chortles with delighted smugness: an expensive phone that isn't even mostly a phone! Imagine the kind of technocratic, free-spending fathead one would have to be to want that! He is becoming a little uncomfortable with the ever-growing percentage of the general population who have joined me in his native city, but he is still happy to soldier on.

However, my point is not his out-of-stepness; rather my point is that my inability to explain to him what he is rejecting. He is rejecting it and then filling in the reasons later. I have failed utterly to convey that the added value is holistic and lies in the totality of the ecosystem. His eyes glaze over when I list the rather large number of functions the "phone" routinely performs for me: check bank balance? Get driving directions? Surf the web to check the hours of a museum? Share links with friends in text messages from which they can go to the same web page? Track my billable hours and expenses? Take photos of friends and family? Take photos of computer parts I need to purchase? Take notes on the train for later access on my desktop? Listen to music? Buy music? Shop for apps for work? Download apps to help me enjoy the museum in which I am standing? It is all too much and too bizarre for him.

So I feel stuck: what benefits he will allow me to present are not enough to explain the actual value of the device+apps+network yet he will not sit through the (admittedly very boring and somewhat peculiar-to-me) actual list. So we have resolved it this way: he is certain that smart phones are expensive toys and that I own one because I like expensive toys, while I shake my head in wonder at what he is missing.

He is not alone in this situation, even in my experience. I have the exact same feeling when I try to explain to IT managers why a given subsystem or middleware is worth building or improving: an endless list of small benefits just does not seem to move people who are not already intimately involved with the given business process. Like a giant jar of small change, which actually has significant value, proposals of myriad incremental improvements sit in the corner, gathering dust.

Smart Planning & Dumb Luck

Managing software projects requires that one make estimates about the level of effort real people will require to solve abstract problems with concrete software.

There are many boundaries in there: abstraction, concreteness, dates, level of effort, human actors and ideas. Business, and much of academia, abhors honest ignorance. Business requires staffing levels, target dates, deliverables, milestones and budgets. The most common approach I see to resolving the abstract nature of programming with the concrete nature of business is reduction-to-known: how is this project like other (preferably successful) similar projects?

In my experience, this approach is good when there is little novelty involved and is more-or-less a crap shoot when the project is highly novel. Many of our competitors handle novelty by assuming the worst and jacking up the price to cover that worst case scenario. We try to break down the project into novel and known and to do basic research so that we can turn the novel into the known as quickly and safely as possible.

But rarely does anyone, including us, talk about luck. Dumb luck. Pure chance. That sad truth is that sometimes you get lucky, sometimes you don't and sometimes you get unlucky.


(A related topic is inspiration: sometimes divine inspiration strikes, making the difficult simple and the giant task almost easy. But I prefer to think that I am divinely inspired sometimes because (spiritual version) God loves me more than He loves other programmers or (secular version) my superior neurology works this way, so I am reluctant to lump this in with luck. Although I understand that lots of other people do lump inspiration in with luck. I suspect that they are not often inspired, but that is another topic.)

I am aware of the "Luck is the residue of design" school of thought (nice context-setting here), and I mostly agree with it, but sadly sometimes I get lucky without regard to my planning or ability or anything else. Sometimes stuff just works out.

For example: sometimes I mention to a friend over a beer that I am stuck on a particular technical problem and he casually points me to just the right person to ask. Sometimes I find that I need to accomplish something and quickly and easily find just the right technical documentation on the Internet. Sometimes I find that an orphaned AC adapter fits a crucial piece of equipment. Sometimes the device driver I need only runs on an old kernel that I happen to still be running somewhere. Sometimes you get lucky: you crush the project, you beat the deadline, you come in under budget, the users are happy and the client is happy to write that final check.

Sometimes you don't get lucky: things proceed mostly as one would expect, with easy parts and hard parts and unexpected gains mostly offset by unexpected losses. These projects plod along and are delivered on time and get paid for and contribute the illusion that luck is not a concern.

Sometimes you get unlucky: critical personnel have family emergencies, supposedly supported devices don't work in *your* server, someone else's piece does not work as advertised or is late, your design is thwarted by the particular deployment environment, the client's server is plagued by gremlins, etc. These projects fray nerves, eat profit margins, sully reputations. We would all avoid them if we could.

(Beware, though, the development team that never has a bad day, or an unlucky project: it is quite possible that they pad their estimates or grossly underpromise. The only way I know to have a perfect track record is to be chronically unambitious.)

How a team or group or organization handles bad luck is very telling. The first hurdle is recognizing when you hit some bad luck and when you didn't. Very few seem able to distinguish between a failure of their team or process and bad luck.

I am not saying that blithely accepting "we were unlucky" is a good way to deal with failure. I am not defending failures of team members or of management or of process: such failures should be evaluated and corrected, either with retraining or retasking people or refining processes.

Neither do I subscribe to the position that good people never have bad luck. It would be nice if that were true, since it would greatly simplify the task of evaluating failure: there would always be blame.

I am saying that realistic appraisal is critical: neither of these simplistic approaches is a good idea.

Having determined that you got unlucky, what then?

Often the response is to avoid repeating this horrible experience by controlling variability. When practical, controlling variable in the software development process is nice, but it not a good primary goal. This is because controlling variability works best with respect to codified processes expected to produce a consistent result. It is a good way to brew beer or make steel. It is not a good way to solve problems in creative or novel ways.

Similarly, if you turn to scheduling as your protection against bad luck, you may be disappointed because if you want on-time over excellent, then you won't get excellent.

Sometimes you get unlucky. All you can do in that case, in my opinion, is pat your team on the back and let them know you feel their pain--but that you expect a return to excellence the next time.

No Substitute for User Input

Whenever I am going to deliver interactive software, I assume that there will be a significant amount of feedback and so I reserver a big chunk of time and energy to deal with that.

I was horrified recently to discover that some programmers feel that I am remiss in gathering requirements, slapdash in my implementation and lacking in my QA. I consider myself to be careful, thorough and to have a high user satisfaction rating.

When I worked through the other programmers' complaints, I discovered that their ideal software development model differed greatly from mine. (And my user satisfaction rating was much higher than theirs, from the same users no less.)

I think that it is instructive to compare and contrast the two models, so that is what I shall do.

I summarize their approach as the classic carpentry mantra, "think twice, cut once." I summarize my approach as "get feedback early and often."

Their model, which is fairly common:

1. Start with a long and thorough requirements gathering phase. The game is lost or won here.
   
2. Write up the requirements into a technical specification. Circulate that spec widely and solicit input.
   
3. Code up the spec as best you can. Write detailed user documentation.
   
4. Test the implementation. Run it through QA.
   
5. Deliver the software to the waiting users.
   
6. Handle the resulting bug reports ASAP.
   
7. Consider the job done and done well. Begin a schedule of releases and updates.

My model, which is certainly not unique to me:

1. Have a longish conversation with the users, mostly centered around them telling me what their ideal solution looks like.
   
2. Create a working prototype as quickly as possible with basic testing only and get feedback. Do a Quick Start Guide as user documentation.
   
3. Process the feedback into work to be done on the prototype. Accepting sweeping changes to the requirements or specs if those changes move you closer to the ideal solution.
   
4. As the prototype iterates through revision, it becomes the first version.
   
5. Over time, issues arise and so there is recoding through the same process.

My basic premises are these:

• Software development is a conversation between you and the users; long pauses will kill that conversation.
  
• No requirements gathering will ever get you more than 80% of the requirements, so plan for real revision at the outset.
  
• Users will happily engage with a dynamic process that produces tangible results

Upon reflection, I could see why my work looks sloppy to them: I put out many versions, I often change my design and implementation, I don't put effort into refining the coding of modules until I am sure that those modules are interested to someone. I don't tend to say "that wasn't in the spec" and I don't complain if things need to change which may seem like weakness or uncertainty to them.

On the other hand, I don't have users who complain that they wait forever for software that doesn't actually do what they need or what it to do.

As is so often the case, each side is not failing at the other's model, each side is succeeding at their own model. I am not impartial, however: I am highly confident that it is better to be one of my users.

Mobile Meditations

We are feeling pressure to enter the mobile space. Everyone else is doing it, so we should too.

As a parent, I tell my daughter that "everyone else is doing it" is not a good enough reason to do something. But as an IT business, we find that our customers sometimes cannot evaluate us on our own merits. Instead, they need proxies to help them decide if we are good at what we do.

Proxy evaluation is sometimes hard to avoidable: I don't go into the kitchen to assess a restaurant; I look at the menu, I assess the ambiance, I rely on word-of-mouth.

So I understand the well-meaning advice that we have to enter the mobile space, to make some of our customers comfortable that we are keeping current and that we are still good at what we do. I understand that more and more people seem to rate a technology company by acquiring that company's free smart phone app and trying it out.

I have even decided to take this piece of advice and to put a mobile app or two on the docket for calendar 2012. I have also decided that discussing this decision will give insight into how we make decisions about what we do and how we do it. This will also serve as an example of an issue that is common in our practice: when something seems obvious at a high-level (create a mobile app!) but is actually quite unclear at the level of action (what kind? how fast? how expensive?)

Specifically, entering this arena raises a number of questions:

• What is the goal?
• marketing
• remove a negative (reassure customers)
• revenue
• demonstrate competence in this field
• What kind of app?
• client for a server? eg Facebook app is a UI to their server
• native app? eg native camera app
• combination? eg some games with a multi-player option
• What business model?
• free? (loss leader, eg banking app)
• freemium? (give away something, charge for more)
• for-profit? (straight up product)
• For which platforms?
• IOS only?
• Android only?
• Both IOS and Android?
• stand-alone, Wifi, 3G/4G, some combination?
• smart phone, tablet, both?
• What market?
• extend our product line to mobile?
• create a separate mobile app, perhaps an add-on to someone else?
• both?
• What development environment?
• native to target platform
• use a virtualization that runs on multiple environments

As our thought process matures, I will return to this topic to record how we decide and what we decide.

In Praise of "Throwaway" Software

A technical person who worked for one of our clients once dismissed my work as "throwaway software." I think that he meant the comment to be dismissive, but I was not insulted or perturbed; I knew exactly what he meant and I was flattered. His comment mean that I had met my goal of providing just what the users needed just when they needed it.

Like me, he had learned to program in the bad old days when development was largely unassisted and laborious. Computer time was the bottleneck, to so we conserved it. We labored over "large" chunks of code, trying to find all the bugs and typographical errors and silly mental mistakes before we dared waste precious processor time to compile our code.

When code is hard to create, hard to change and likely to be in service for years, it makes sense to move slowly and carefully, to write for the ages. "Nothing ever gets thrown away," we were told. "Anything you write is likely to be recycled and to live forever, so every line should be immaculate and immortal."

If you don't remember assembling programs into punch card stacks and then putting those stacks into wooden trays so sys admins could load them up, then you are going to have to take my word for it: coding used to be a cottage industry: we lovingly hand-crafted every line. We were thoughtful and careful and only did Important Things with the Mighty Mainframe. We were very aware of our High Priest status and we did not want to anger our computer/deity or waste our precious opportunities to interact with it.

These days, thank God, the balance has shifted dramatically: computing power and disk space are cheap, so development is less about writing fantastic code and more about refining your software so it does exactly what it needs to do. The computer helps with with syntax-aware editors, chromacoding, ease of compilation or interpretation, on-line help, etc.

With the advent of "wrong sizing," there are usually too few workers doing too much work. As mentioned before, human attention is at a premium. This includes both the humans writing the code and the humans using the code. This means that developers need to do what makes them productive in terms of writing code, and software needs to do what makes users productive at their jobs.

Often, this means writing special-purpose, "throwaway" tools to help specific users do specific jobs. If you are one of the users, struggling to do an important job, you are not going to like the description of your urgent need as "throwaway." If you need support for "only" a few months, after which the tool will never be used again, so what? You still have the need and computers can still fill it.

(To me, an "app" does a fairly high-level job in a fairly complete way, while a "tool" does a very limited job in a possibly unsophisticated or unpolished way.)

I am struck by the fact that builders don't have this hang up: special purpose scaffolding is not "throwaway building." No one says "what is the point of building concrete formers when you are just going to tear them down later?"

I watched an expert put up a shelving unit for me using a temporary scaffold, which was eye-opening. He was slight and elderly. I was bigger, stronger and younger. I offered to help him put up the unit. He was polite in his refusal of my help. I stuck around to watch because I was baffled. He eyed the unit and wall space. He threw together a scrap wood frame. He and I put the unit on the frame, at which point the unit was right where it needed to be. He screwed it to the studs in the wall with two cordless drills: the left hand drilled pilot holes and right hand used a screwdriver drill bit to sink the screws. With astonishing efficiency, the unit was up. He then removed the frame and, with the screwdriver drill running in reverse. disassembled the frame. Voila! a throwaway frame made the job almost trivial--assuming that one is skilled enough to throw together such a thing, which I am not.

I recently created such a tool to help one of our clients debug an interface. I knew going in that once the interface was working properly, the tool would be of limited use or interest. I also knew that without the tool, without the window into what was going on, the cycle of "we never got it!" being answered with "well, we sent it!" would go on forever.

As is so often the case, in the course of refining the tool and chasing down what the tool revealed, we found that there were many minor issues which appeared (falsely) to be a single, serious issue. Everyone was mostly right. No one was evil or stupid or uncaring. Once the tool gave us a shared context in which to have conversations, months of bickering were settled in days of rational, shared-reality-based email exchanges.

Is the tool "throwaway" software? Perhaps. Was it quick and easy to create? Yes. Does it provide excellent value for money? Absolutely. In this day and age, is "throwaway software" an insult? I would argue only for those living in the past.

(Note that I am not advocating writing buggy, sloppy code. Quite the contrary: weighting the effort toward refinement and debugging, in my experience, produces a lower bug count and a better user experience.)

My advice is to get out there and start whipping off solutions to small- and moderate-sized problems. Your users will thank for it, even if your software only lives as long as it has to.

Rigid Engineers, Obsessive MBAs

I feel a good, old-fashioned rant coming on. Just for balance, I will combine two rants: Rigid engineers and obsessive MBAs.

RIGID ENGINEERS

Engineering is a great discipline. I like it. I try to adhere to its basic tenets. I feel that engineering has the following strengths: it is rigorous, it is regular and it focuses on the relevant.

Bridges tend to stay up. Planes tend to fly. Ships tend to float. If you have a house or a car or a smart phone, you owe the engineering discipline thanks.

Software engineering is a little bit different: our building materials are abstract and our rules are a bit fuzzy and our adherence to our rules is not guaranteed.

Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization.

However, my rant today is not about loosey-goosey software engineering, but rather its opposite: inflexible software engineering. The discipline of engineering is not good when there is no good rule and some software usage requirements create a very sloppy execution path. Instead of a measured, graduated response to the complex set of requirement, instead of a long chain of events from messy reality to complex model of the messy reality, the rigid engineer gives us a clean abstraction. Even if the clean abstraction does not, in fact, support the actual business process to be automated.

I have had to deal with engineers who actually raised the following objections to proposed designs or changes to their beautiful cathedrals of structured code:

• "That would break my parser"
• "That would require a goto"
• "That would importing a large, messy library"
• "My structure can't do that"

The subtext is always the same: "I want reality to be changed to suit my model." This often expressed as "the user can do {long complex sequence of instructions} to get the same result. There is no need to change my code."

To me, this is the tail wagging the dog: software should do what it needs to do, first and foremost. It is the job of the engineer to make that happen. It is not the job of the engineer to change the scope of the task to make the software simpler, unless the scope is beyond the ability of the implementor. In that case, the implementation folks should say that and try to work with the users to find a compromise. Decreeing that some functions are a bad idea is not what I mean by "compromise."

This belief that if your implementation cannot support it, then the request is bad and the implementation is good is related to the "great code is not great software" idea I touched on before, here.

OBSESSIVE MBAs

A colleague drew my attention to this great article:

http://www.forbes.com/sites/adamhartung/2012/04/20/sayonara-sony-how-industrial-mba-style-leadership-killed-once-great-company/

This article is a study in how classical MBA management has hurt Sony. The article describes a situation that I find increasing common not just in consumer electronics, but also in IT. (I have ranted about other manifestations of this, notable here.)

This article describes the typical MBA focus on operating efficiency, a la Demming. This focus is great, short-term: assuming that what you are doing today is what you need to be doing tomorrow, then finding better ways to do that is a great idea.

But in technology, figuring out what you should be doing tomorrow is often very important. Operating efficiencies gained with obsolete technology are often smaller than operating efficiencies to be gained by using current or even leading-edge technology.

Taking chances and taking risks as you figure out what it all means to you pays real rewards.

But classical MBA management teaches managers to be wary of the new and to concentrate on the current. In an industry where 18 months is a typical development cycle, a three year plan to increase efficiency will be two cycles out of date by the time it is finished.

I understand that operating efficiency makes sense. I realize that most organizations cannot handle constant flux. I appreciate the need to plan and to budget. But I am horrified at the ever-increasing tendency of the managers we meet to assume that all technology is pretty much the same and that three-to-five year roll-out plans make sense for software, with "freezes" while the implementation is done and evaluated. No wonder so many companies are out of date and out of touch with IT.

Programming by Any Other Name

These days it is hard to find even moderately sized systems which do not claim to offer rule-based options for configuration. These systems claim to provide the flexibility of encoding business rules without the hassles or requirements of actual programming.

Ha! say I. Call it what you like, computer programming is computer programming.

Juliet:
"What's in a name? That which we call a rose
By any other name would smell as sweet."
Romeo and Juliet (II, ii, 1-2)

Inspired by the great Shakespeare I say that programming, by any other name, still smells of sweat.

Note that I am not saying that rule-based systems are not useful, or powerful, or neat-o. I am saying that translating real-world polices into absolute statements in some rule-definition language is not a common skill and is so similar to most kinds of computer programming as to be indistinguishable.

Dress it up however you like: if you are not good at programming you likely will not be good at writing rules, let alone debugging rule implementations or grasping the cumulative effect of a large rule set on a large and complex application.

I suppose one could divide up computer programming into "predicate logic" and "computer stuff" (RAM, file I/O, databases, communications, etc). Using this division, I can imagine a person who excels at predicate logic but has no feel for the computer stuff, making this person uninterested in computer programming.

If such a person ended up as a business executive instead of a formal logic academic, (or APL programmer) such a person might be quite good at writing rules. That person might even be able to debug her rules, to cope with the often-wide chasm between the abstraction presented by such systems and the concrete reality presented by the working environment. But how many such people can there be in the world? Certainly not several in every moderately sized company in the world.

So when sales people tell you that, at last! you free of the tedium of dealing with IT professionals, finally free of the tyranny of computer programmers, stop and ask yourself this: are you really ready to eat your own cooking?

As every even junior programmer knows, predicate logic in action is rife with  unintended consequences which are the inevitable-but-unforeseen results of rules as applied to the real world.

My personal favorite example of this lateness and snow storms. Most or all of our clients above moderate size use Kronos Time & Attendance products. Most or all of our clients, in the frenzied enthusiasm of the original roll-out, go overboard with the rules, often with amusing unintended consequences. One client decided to implement a "better never than late" policy: if its employees were more than one hour late for work, they could not swipe in at all. This was intended to enforce punctuality. I do not know how it worked in that regard, but I do know that when I arrived mid-morning after a large snowstorm, I was greeted with a stream of employees leaving work: the snow had delayed them and they could not swipe in; unsure that they would be paid for the day, they were going home. Not much got done that day, but what did get done was mostly done by upper management, who were exempt from that rule.

In this case in particular, I like the "silver bullet" trope because even though there is no magic solution to the problem of translating human policy into machine-readable form, the problem can, werewolf-like, sudden get very ugly very quickly under the right circumstances.

So see a doctor for medical problems, go to an accountant for accounting and get your rules from a machine-readable logic expert. Or don't, but at least go into your decision with your eyes open.

Almost Never Say Die

In designing, creating, deploying and supporting information systems, I follow the military model: I call lower-level decisions and tasks "tactical" and higher-level ones "strategic."

Like many (most? all?) successful people, I hate to fail and I don't do it very often, at least at the strategic level. In fact, I can't remember ever truly failing at the strategic level. Failure at the strategic level is usually catastrophic: entire projects which are never used, technology that just doesn't work, solutions that either do not solve the problem or solve the wrong problem. Avoiding strategic failure is usually matter of either knowing your problem space or recognizing early that your solution is not right.

In a sense, I fail at the tactical level all the time in the sense that I try out approaches and reject them if they do not work sufficiently well. I very rarely, if ever, simply throw up my hands in despair and tell the users to live with it.

(I do sometimes tell the users that their boss won't pay for the fix, but that is a different story, even if that story looks the same to the poor users.)

The key to useful failure, as opposed to disastrous failure, is a sense of what should be. How do you know that this job is taking too long and that a different approach is warranted? Because you have a sense of how long the job should take.

How do you know when a restructuring, an expensive, time-consuming, bug-introducing restructuring is required in order to support an apparently trivial upgrade? Because you have a sense of the present time and energy the lack of restructuring is causing and the future pain that the restructuring will avoid.

For example, I threw in the towel yesterday: I was supposed to complete a relatively minor upgrade to one of our apps. The implementation was complicated by the fact that the client had insisted that databases were too big a hammer to use to crack the walnut of configuration, so this app uses text files to store records and support a simple Web-base editor UI. I realized that having extended and re-jiggered this code for years, this final upgrade was a bridge too far: a job that should have taken an hour or two was going to take much longer. In fact, after an hour, I had just about figured out my approach.

I was also behind on delivery because I kept avoiding this project because I knew, deep down, that it would take too long and have too many bugs in the initial implementation and that it would, by my standards, be a failure. And I hate to fail.

Yesterday, after my planning and before my coding, I had to admit to myself that the right thing to do is to rewrite the entire configuration handling, basing it on...a database, as one would expect. This will make the editor easy--and easily stolen, eh re-used, from other parts of the same project. This will make the current and future changes easier, faster and safer. This will make the end user experience better than it is now. This will make the apps which use the configuration information simpler and more dependable. It is the right thing to do.

I will still fail to deliver this upgrade in what I consider to be a timely manner. But I will succeed in the upgrade and I will make the app better, faster and more robust.

One of the joys of getting older is the ability to recognize tactical failure, or impending tactical failure because once you recognize it, you can avoid it or side-step it.

Here I swerve into the uncomfortable area of ageism: the flip side of our industry's love affair with youth and energy over age and experience is a frequent lack of strategic sense. I cannot count the number of times I have had to tell a young programmer, who has proudly shown me his implementation of something, "it took you 12 hours to do a 4 hour job: you should have come to me for help instead of plowing ahead." I don't care that he (always he) stayed at his desk for 12 hours. I don't care that he did it "himself." I don't care that he feels that his implementation is special; alas, I only care that time and energy was wasted.

I am middle-aged: I no longer am willing to code for 14 hours at my desk, bullishly pounding away until the tactical job at hand is done. I haven't tried in years; perhaps I simply cannot do it any more. I hope never find out, because I am now sure that only strategic failure requires such tactical sacrifice.

On the other hand, I also no longer end up having people, when I am done with my death march, asking me "why did you spend all that time on that job? Couldn't you sense that something was wrong?"

I can sense that something is wrong. I can even admit it. And that makes all the difference.

Prose Before Code

I am going to criticize my fellow software developers in particular and engineers in general: I claim that as a group we write boring, confusing, rambling prose.

I realize that I am inviting criticism of my writing by doing this and I welcome the scrutiny: my claim is that writing adequately is within almost anyone's grasp and that I follow my own advice. Alas, if you find this post boring, confusing or rambling, then I have a problem.

I will confess at the outset that while I make my living writing software, I like to write expository prose. I always have. I like writing speeches, I like writing toasts, I like writing presentations, I like writing white papers. You name an expository prose form and I have probably written at least one of it.

Of course, enjoying pursuing an activity is not a reliable indicator of how competently you pursue that activity. Sad, but true: you can love doing something and still be terrible at it; witness my attempts to play Mario Kart on the Wii with my daughter. How can I be that bad? It is an enduring mystery. As my daughter often asks, "why can I beat you when I can't even drive yet?"

Sadly, unlike a video game, writing often does not provide adequate feedback, mostly because we do such a bad job of soliciting it. (Hint: asking a colleague to review an awesome thing you just wrote awesomely does not provoke honest feedback.)

I am even interested in the teaching writing; I don't it myself but I know many who do and I have had many enjoyable conversations with professionals about the trials and tribulations of teaching writing at various levels.

While I am in a confessional mood, I will admit that I was both an English major and a hardcore computer programming student in college. I am, unsurprisingly, also a big fan of reading. I interested in critical reading, recreational reading, skimming technical documents, serious reading of literature, referring to references and even reading advertising copy.

What I do not generally enjoy reading is expository prose written by engineers of almost any stripe, computer programmers included. My prejudice is shared by all the teachers of writing I have ever queried, from the ardent high school teacher to the burned out English-as-a-second-language coach who had just retired from helping a major American car manufacturer get more-or-less comprehensible prose from foreign automotive engineers.

I am so baffled by the extreme badness of so much engineer process for the following reasons:

1. Being a terrible writer takes ambition. You have to go for it, you have to attempt the bold and broad, the complex and the complicated. If you stick to short sentences and use standard vocabulary, you may well be boring, but you won't be cringe-inducingly awful.
2. Being a terrible writer requires that you flout the rules, that you go your own way, that you find your own special drummer and follow him or her without regard to where he or she takes you.
3. Being a terrible writer usually requires that you ignore your reader and do whatever you want to do, that you make assumptions about what the reader knows, or finds amusing, or finds clever.

But engineering and its training are all about the rules. No one expects you to rediscover the laws of Physics or the principles of circuit design or best practices in software development. Instead, you are expected to know these already and adhere to them. So why do so many engineers feel that picking up a pen, or sitting at a keyboard, is license to just go for it? After all, the compiler or interpreter or circuit board or laws of motion have no sense of humor whatsoever. We should be used to leaving the flourishes for our hobbies and our social interactions. (I know, I know, we are not a group with whom most people rush to party.)

When I corner a hapless professional writing teacher at a dinner part and run through my standard rant, the most common explanation offered is the student empowerment movement in American education. Be free of the rules! Express yourself! Be original! Let your natural talent soar! Find your voice!

What twaddle. Most of us have precious little natural talent; fewer have a pleasant writing voice that appeals to most people.

Furthermore, the forms and conventions are very effective. They give your reader a sense of orientation, of familiarity and of confidence that they know what is coming. Do I want to be startled by the originality of a white paper on how to make database calls from a given programming language to a given database management system? No, I do not: I want to acquire the desired information quickly and effectively. I have searched for this kind of information many times before and will do so many times again before I retire: excitement, suspense or high style are neither desired nor required. Just the facts, ma'am.

For example, the Unix "man page" format frees the writer from having to figure out how best to document a given function and instead lets the writer concentrate on writing the actual documentation. On the other end of the equation, the format means that the reader knows what is coming and how best to read the document.

I have been guilty of this desire to run amok myself: the first time I wrote a business plan, I was appalled at how outdated and stupid the classic form seemed to be. I knew that I could do better. I yearned to do better. I grit my teeth and half-heartedly adhered to the form and the plan was not a big hit with the investors and possible recruits at whom it was aimed.

I asked a friend who is in the business of reading business plans for some advice. Instead of undertaking a critique of my plan in particular, he gave me a sense of the audience in general. He said that he had a stack of a couple of hundred business plans on his desk, which he was supposed to quickly whittle down to a dozen or so. That dozen or so were to be passed along to the next, more senior, reviewer, and so on. In order to review hundreds of plans, the plans had to adhere rather strictly to the format: non-standard plans were usually tossed aside immediately; very very very occasionally, the non-standard plans were so awesome that he read them later, retrieving them from the discard pile. But almost always, the non-standard plans were removed as part of the first pass without really being considered.

This underscores a painful lesson I learned early on in my writing career: very few of us have readers who are obligated or deeply motivated to read whatever we write. Most of us have readers who will plow ahead only so long as they are getting more out of the writing than they are putting into it. One of my secondary school writing teachers used to put a red line in the left margin. I asked him why he did that and I will never forget his answer "to mark the point at which I stopped reading."

Sadly, many engineers fail even when the deck is stacked in our favor: we often write documents other people feel obliged to read: manuals, implementation notes, etc. And still we abuse our readers to the point where they give up part through, leaving them to flail with whatever piece of technology the documentation was supposed to illuminate.

There are many good books on writing. There are many tips and tricks. I will not attempt a mini-recap here. Instead, I will beg my fellow engineers to seek honest feedback from readers, to consider readers and to find a set of rules to which they can adhere. Just because we can get away with self-indulgent and awful prose does not mean that we should.

Modern Modularity & Iterative Development

I recently wrote an app in about two hours: a highly tailored, very powerful tool for a clerk. And I did it as a Web CGI app using straight Perl without the benefit of a rapid application development environment or even a framework like Mason.

Hurray for me. Except, of course, I didn't really do that. It is true that I wrote the app in about 45 minutes and then spent a little over an hour in a tight user feedback / iterative development cycle, resulting in the finished product. But this is the movie version, the marketing hype, the storyline that ignores all the boring ground work that went before.

It would be more accurate to say that I spent about dozen hours over several weeks on various stages of this project: still fast, still cheap, just not spectacular (or utterly unbelievable).

From my perspective, the process went like this:

1. The client defined a problem: clean up a server's disk area by removing redundant files stored by a system of ours.
   
2. I defined the reconciliation algorithm, using md5sum to confirm that files exist in both places. Along the way we encountered some horrific special cases, but that is par for the course in the real world.
   
3. I wrote a audit program to put a random sample of these files in a web page for review; our clerk reviewed several batches of these files spread out over the entire repository. We found that the issues were highly concentrated in time windows, so we tailored our approaches to fit the eras that we found.
   
4. Based on her feedback, I reused much of the audit program to create a web-based UI to support her review and I wrote a couple of small helper utilities to automate the review process that the clerk had used. She confirmed her earlier era-based findings.
   
5. I added a database table to hold the information that the clerk was tracking on her paper pad and which she had figured out was what she needed to know. This allowed us to break down the repository into manageable chunks and to start tracking the clerk's progress more accurately.
   
6. I wrote a daemon to do prep work in the background and another one to automate as much of her review as I could. This cut the number of items requiring human review by 90% and allowed us to start automatically correcting that 90% while she worked on the ugly 10%.
   
7. Once all that was in place, I wrote the final app, a UI to review the contents of the database as set up the by daemons and to add editing functionality to support saving the results of the review. Then I added a few links to those helper apps to support the reviewing and I was done.

This is how I write good user-intensive software: a design phase which moves quickly to practical applications; we figure out the ideal process and the software trails quickly behind until we have a system that we all trust and that does the job.

Note that my utilities and previous code are highly reusable in both senses:

• I can use the code with little or no modification in different apps, which means that I only have to develop and validate once but use many times.
  
• The stand-alone web helper apps are easy to string together to enrich the environment provided by the main app. This is an homage to the Unix "string of pearl" philosophy which I find works so well at the O/S level.

In theory I should get both of these benefits from the Object Oriented programming paradigm, but in practice I don't seem to.

In this day and age, the web technology allows us to have separate-but-easily-integrated pieces and modern programming languages allow us to reuse chunks of code easily so there is no excuse for not producing polished and power apps rapidly and cheaply.

When You Can't Read The Code

As the large enterprise solution becomes more and more common, I find that the keepers of such systems seem less and less interested in putting in the time to become domain experts.

As IT systems consultants, we are often interacting with these enterprise solutions. Some percentage of the time, we encounter behaviors in these systems which we find unexpected or undesirable. These behaviors may or may not be bugs, but we feel that we should understand them before we ignore them or program around them.

Back in the day, when more apps were homegrown and more systems were provided with source code licenses, the people we asked about unexpected or undesired behavior were the client's own technical people. If they did not know the answer, then they could find out by reading the code.

Increasingly, we find that our client contacts are either non-technical people or out-of-date technical people. Only a small minority of contacts are technically competent, helpful providers of deep background and technical information. We often have to bypass the client and go straight to their large system vendors because there is no local technical expertise.


To our dismay, it is our technical bretheren, our peeps, who are the hardest to deal with: now that they cannot read the code, they tend to either shrug off our questions or give us shallow guesses off the top of their heads. After all, if you can't be a hard master then why not just give up?

To our delighted surprise, the non-technical people are generally easier to deal with because they know that they don't know and they look for the answer. They will actually contact the vendor for information, or find documentation, or give us the benefit of the user's experience.

While we don't like treating systems as black boxes or as puzzles into which we pour various inputs and monitor the outputs, at least these methods actually turn up results. Getting half-baked guesses from formerly expert techies just wastes our time as we discover just how half-baked the guesses are.

Just recently I spent an afternoon running a series of experiments as we tried various inputs into a large system to see if we could figure out how to accomplish a particular upgrade for the user. We started with a half-baked guess from a supposed expert techie, proved that this techie was wrong about how the large system worked and got down to the business of figuring out to do what must be done.

It seems to me that technical people have to be able to operate in the absence of hard data. My checklist for answering technical questions goes like this:

1. Query experts if you can
2. Read the code if possible
3. Read the technical documentation if there is any
4. Query super-users if you can
5. Read the user documentation if you can
6. Query experienced normal users if you can
7. Write code to grok databases, logs, etc
8. Try to reason from first principles
9. Guess because there is absolutely no other option

And remember that when talking about technical matters, which are not generally matters of taste or subjective opinion, trust but verify.

Hugs not Bugs

Ok, this week my posting is a self-pitying rant, pure and simple. The topic: asking for help from software developers.

For some reason, our industry has adopted the following value: if you need help, too bad; if you found a bug, we will do all we can for you.

At least, this is how we seem to be perceived. I spend much of many of my days at work dealing with accusations of incompetence, which accusations are actually poorly-disguised pleas for help. Whenever a client or user needs my considerable analysis and analytic skills, they ask for help by accusing me of having bugs or poor design.

Yes, I have bugs and yes, I don't have a problem with bug reports. I am highly confident that I can fix any bug and I have few enough of them that I am happy to fix whatever needs fixing. I am not talking about bug reports: I am talking about something else. Here are two examples from the recent past; one is from yesterday, in fact.


Case 1: Just Fix It!

A client reports that a system of mine which is designed to create orders is failing to create orders. She knows this because my system comes complete with an built-in audit module that told her so. My system is at the start of a very long chain, going through another department in her organization and through two other vendors' systems.

• I confirm that my system generated a request to create an order, which request was acknowledged by the next system.
• The bug report is repeated, this time with a slightly ruder, slightly more accusatory tone.
• I confirm, by looking at logs that are not my own, that the request did not reach its final destination.
• I ask the intervening department to confirm that the request went through their system and out the other side.
• I confirm that the final destination has two interfaces through which the request must pass, and that I only looked at the second of those.

I am slated to continue tracking this down, through other vendor's software as configured by other people, not because this is in any meaningful sense "my" bug but because I am good at this and am being paid to do it. They need my services, I am happy to provide those services: why does the request for service have to be framed so negatively?


Case 2: Its Not Working!

In one of my professional identities, I provide technical support for a line of technical products which I designed. This gives me contact with the retail world and a sense of how my work is actually used in the real world. Most of what I learn is terrifying and makes my consulting clients seem like ever-pleasant geniuses.

I recently fielded a "bug report" that a four-year-old version of our software had mysteriously stopped working and that if there was a patch, the customer lacked it.

This is, of course, absurd on its face. I directed tech support toward a tech blog posting I had made some time ago about debugging configuration files, since this was obviously a case of misconfiguration.

The reply, as forwarded to me, was that the data had not changed, the configuration therefore was right and that the fault lay with the software.

This was ridiculous, but arguing with someone who is either this ignorant or this belligerent is a waste of time, so instead I used our excellent configuration debugging tools, which are available free for download to any of our customers, to find and fix the small issue with her configuration: Voila! it all works--or rather, it was working all along, but now her configuration matches her data.

Naturally, it turned out that this particular user was one of the few to send a thank you note, a nice thank you note, when she got the resolution. So I try to remember that not every difficult user is a troll: some of them just need help and they apparently don't know how to ask for it.

How Much Technology Is Enough?

I am generally a minimalist, at least when I am designing or implementing information systems. To me, minimalism means doing it all, but not to excess.

So when I go out to dinner, I will have a steak, a baked potato, a spinach salad and some red wine but will have a small steak and a reasonable amount of wine.

I realize that many people would quibble with that definition of minimalism, so instead in this post I will use the term "optimal" to mean "everything you need but nothing that you don't need."

While I feel that the finished product should be optimal, I do not feel that a project's resources should be only just enough. Instead, I believe in what a colleague once referred to as "cheap insurance." By this I mean that I believe in providing an excess of whatever critical resources can be cheaply and easily procured.

For example, I often buy a USB external disk (which we call a "can") or two just for the project to make sure that disk space and back up will not be a problem. USB cans are cheaper than having a project delay or a project  disaster. Similarly, I keep a few unused computer system units around because you never know when you will need a special purpose machine or want go provide a contractor with a machine.

All this keeping of stuff makes me s bit of s techno pack rat; although the difference between techno pack rat and project savior is often timing or dumb luck. And shelving is cheaper than failure.

Since we often help customers get off of old platforms, I have another reason (or, according to some people, rationalization) for keeping dated technology around: the ridiculous pace at which the high tech world changes.

Even applied technology tends to be very much of its time: often the dated peripheral has to match the dated system unit and match the dated driver running under the dated operating system. All too often trying to use a shiny new peripheral on a mature system simply does not work; worse, sometimes it works to some extent, sucking up your time as you fiddling a vain attempt to get full functionality.

Another reason that I hoard old tech is that I have a deep respect for working systems. I know from painful experience that every time you fiddle with a working system you risk having a systdm that will never work again. Am I a nervous Nellie or a scarred veteran? Opinion vary, but I will stick to my cautious ways. I like taking chances just fine if I have a fall. Ack position.

Not only do I fear that working systems are only working until you fiddle with them: I also fear that projects of more than trivial scope never go as planned. I know that contingency planning is good but not enough, so I want more: I want options. I want flexibility. So I need lots of extra stuff in order to be able to suddenly decide that the system would work better if there were two servers instead of one, etc.

I find that debugging distributed information systems sometimes requires creating a parallel universe. If the bug or issue or misconfiguration or mismatch is deep enough, you need a testbed in which to try changing fundamental aspects of the system. Sometimes I need a consult from an external expert, in which I want to be able to deliver a working prototype to them while still having a working prototype in house.

So I find that in order to deliver enough product, I need way more support than I expect. However, I notice that the same people who tell me I am going overboard on the technology resources for projects are often also the people whose output is lacking that final polish, those extra features that distinguish adequate from good and good from great. Shelving is relative cheap: I am going to keep pushing for great.

(Although even I have my limits: I have 8 bit NICs and CGA video cards I would be willing to part with at a great price.)

Hard vs Soft Mastery

Some years ago, as part of a discussion of computer programming styles, a colleague introduced me to the concept of "hard master" versus "soft master." Be warned: I am emphatically not making any value judgments here: I do not believe that hard masters are smarter, purer, better or more detail-oriented. I do not believe that soft masters are failing at being hard masters; I do not believe that a good hard master can do anything a soft master can do, or vice-versa.

Enough about what I don't mean: what do I mean? A little Google magic got me here, the horse's mouth: Feminism Confronts Technology by Judy Wajcman. Read the selected section for details, but the summary is this: the hard master feels he has to know all the details when using technology while the soft master knows how to get what she wants.

The use of gender-specific pronouns is not accidental: frequently hard masters are male and soft masters are female. Note that while there is a significant pro-male, anti-female bias in computer programming, this is not a bias from which I happen to suffer. I have worked with women for my entire technical career, and have found female technology interaction different but not inferior. In fact, I have found that this discipline has room for, and needs both, kinds of interaction. As a general rule, I want my bit-bashing systems software from an anti-social grumpy guy and I want my application software from an empathetic, flexible and cheerful gal.

I understand that these generalizations are not universal truths, that there are grumpy women who are antisocial and mathematical while there are cheerful men who are sensitive to how technology feels to use, and all possible permutations in between.

But I do not believe that all permutations are equally probable. I find distinct trends: men tend to define technological success and men tend to be hard masters. Women have their own technological approaches and these approaches tend to be deemed inferior by men, mostly men who cannot perform these "soft" tasks themselves.

Given this observation, I was not surprised at a recent recommendation of related reading from another colleague: an article from the Harvard Business Review that considers the assertion that women make your technical team smarter. I am sure that is true, at least for my team.

In the interests of full disclosure, I will admit to being a middle-aged man who thinks of himself as a designer, as something of a non-combatant in this fight. My attitude is that I can code anything I have to, but I am not a coder. Opinions of my coding by coders varies, but mostly I think that both hard and soft masters have reason to despair at my code. So I claim to be of neither camp: I need them both to get my designs realized, unless I end up having to code one or other kind of task myself. Which happens all too often in these recessionary times.

I find the hard versus soft dichotomy in server software versus client software. For example, configuring a web server is a heavily technical task which requires a deep understanding of networking, process management and security. By contract, creating a web page is also web-related, but has utterly different requirements, including a flair for graphic art and an understanding of usability.

I also see this division in the back end (data storage & retrieval) area as opposed to the front end (data acquisition through user interaction) area. Figuring out how to store data effectively is a job for a hard master who wants to think about file systems, database formats and the interaction of caching and performance. Figuring out how to get users to enter data quickly and accurately is about understanding how human beings use software.


From my perspective, there is trouble brewing for the hard masters. It is my observation that as abstraction rises mastery softens. When there was only simple hardware, Unix and ANSI C, someone could be a hardware expert, a firmware expert, a Unix system software expert and complete master of their own code base. Now, with server clusters and interpreted languages and database interfaces and web servers and programming environments embedded in web servers, I just don't see that as a viable or desirable goal. How do hard masters get anything done these days? They must have to restrict themselves to rather small areas of expertise.

Not only is abstraction on the rise, but I see a very real trade off between productivity and mastery which abstraction provides. Women who are happy to cruise along at the GUI-to-create-GUI level, such as Visual BASIC, kick my ass in terms of how fast and how custom they can make apps. Who cares how deeply they understand what underpins that abstraction? (Until that abstraction bites them in the butt; more of that anon.)

There has been a shift in the balance of power between apps creators (soft masters) and systems engineers (hard masters) as time has gone by. So far as I can tell, app programmers now rule: users understand what they do (create apps), economic buyers are willing to pay for what they do. Also for systems engineers, I find that users have no idea what they do, economic buyers don't relate to them or their work and organizations feel that they get the behind-the-scenes stuff for free, as part of buying apps, or smart network hardware, or whatever.

When I started out in the early 1980s, app programmers were the bottom of the ladder, while systems programmers did the real work, got paid the real money, and wrote apps whenever they felt like it. As a designer, I find that I no longer have to start by getting hard masters to buy into my design: in fact, I find that most of my gigs come from soft masters coming to me so that I can create a framework in which they can create the apps that people want. No one is interested in the lowest levels: is this MS-Access to a local database, or MS-Access as a client for a database server? Who cares?

Of course, users should care: bad infrastructure makes for a bad user experience. But users don't care about boring old infrastructure and systems engineering because most of the time they can afford not to.


All this does not bode well for the hard master. These days, I see hard mastery in demand only in those relatively rare instances when the abstraction fails or is incomplete. When you need a DLL to extend the Microsoft desktop, or a shared object to extend the Perl interpreter, you really need that. But how often does that come up? When I started out, we had five or six programmers, all male, two of whom were apps specialists. Over time, we have tended to add apps programmers, who tend to be women, and we have moved to consuming our hard mastery as a pay-as-you-go consulting commodity: we don't need full-time hard masters anymore.

This makes me a bit nervous: will the hard master be there in the future when I need him? But for now, I am busy being part of the problem: I need more soft masters and fewer hard masters I don't see that trend reversing anytime soon.

Impure Implementations

I am a big fan of using the right tool for the job. When it comes to carpentry or surgery, this concept seems obvious and well-understood and usually followed. But it comes to large IT projects, I find a real tendency in large organizations toward trying to solve all problems with whatever single tool they have blessed. Let us call this tendency the Silver Bullet Assumption.

Many organizations are Visual BASIC shops, or Visual C++ shops, or Python shops, etc. This baffles me: there are many excellent tools out there, but there is no tool that is great for every aspect of a large project.

I find that large projects usually have most or all of these aspects:

• an inbound interface for acquiring data
• a collection of data processing functions
• a way to store the data
• a user interface (UI) to view the data
• reports and exports to send the processed data down the line
• an outbound interface for sharing data

Thus most of these projects have some systems-level requirements, some user-level requirements, some low-level data processing and some high-level data manipulation. What single tool is oriented in all these directions at once? If a single tool did encompass all these aspects, could it do so in a consistent manner, or would it be a wrapper on a wide collection of disparate parts?

Even if such a tool existed, who would use it? Someone who understood all those different domains?

In our consultancy, we have a break down that I think is pretty common, or used to be: we have systems people (who work mostly under Unix) and apps people who work mostly under Windows and Web work lies somewhere in between.

We use MS-Office or web pages to provide UIs on the desktop, web pages and thin clients to provide data entry on the floor, Unix servers to provide print service, file service and web service. It is hard to recall a project of any scope that did not cross these boundaries.

We are constantly asked questions about implementations which assume that everyone does everything: the Unix systems programmer who is supposed to know about MS-Access apps and vice versa. When we push back, we find that many organizations have the notion of "programmer," or even "systems programmer" versus "applications programmer" versus "server guy" but all these programmers are using the same environment: Windows or Unix and it is mostly Windows.

Clear as we are about our design philosophy, even we occasionally have requests for "pure" implementations, with the hope that if the technology under a large project is consistent, that large project will be easier for local IT to understand and support.

But this is often a forlorn hope: if your people do not understand bar code grokking or TCP/IP-based protocols, it very likely won't help if the thing they don't understand is implemented in a familiar technology. AT worst, they will have a false confidence that will lead them to fiddle where they should not fiddle.

(I speak from bitter experience, of course. Ah, the support phone calls which start by saying that some of our technology does not work and end with them admitting that they "fixed" something else right before the mysterious new failure began.)

I just don't buy the premise, that being fluent in systems, apps, networking, infrastructure and databases is a reasonable expectation, let alone the usual case. You know that you need network people, desktop support people, server people, etc. Why do you think that they all should be working in the same environment? What does that even mean, when you think about it: how is a desktop app like a print server?

This illusion of the benefits of purity is encouraged by vendors, so I suppose the customers are not really to blame. The first time I laid hands on Oracle, lo! these many moons ago, I was stunned at all the special-purpose configuration file formats and languages I needed to learn in order to tune the installation. But the client thought of themselves as a pure Oracle shop. This is like saying that all of humanity is part of a pure human language shop--we just use different flavors of language.

Very recently, I worked with a system that was billed as all Windows, all the time. Except that when push came to shove and I needed to debug some of its behavior, I come to find out that the core was a Unix server running a ported COBOL app. Egad! Knowing that it was COBOL through which the data was passing made debugging that systems interface much easier, by the way.

 Why tell the customer that they are buying a Windows app running on Windows servers, with some kind of remote back end? I don't know: it must be comforting to someone somewhere.

I prefer to be more upfront with my clients: I will use whatever technology will get the job done, with an eye to accuracy and speed. I want to save my time and their money. I try to use technology that they already own, but I cannot guarantee that--unless they want to pay extra; often LOTS extra.

If I have to use MS-Access at the front, FTP in the middle and Oracle on the back end, then so be it. I find the choice between requiring minimal maintenance, but making local IT uncomfortable, and requiring lots of maintenance, but making local IT (probably falsely) confident, an easy one to make.

Just last month, we shut down a system of ours that had been in continuous operation since early 1984. That's 27 years of service, with an average of under 10 hours of attention per year. This system's impure implementation made local IT nervous, but it also allowed us to adapt to the dramatic infrastructure changes over that time. In the end, it was time to retire it: a 16 bit environment runtime environment under a 32 bit operating system running on a 64 bit architecture is a bit baroque even for me.

So while nothing lasts forever, I claim that the concept is sound: until there is a single, simple, all-encompassing technology, use what makes sense, even if the final product has multiple technology environments under the hood. There is no silver bullet and there never was.