IT Ecosystems or Many Small Benefits

I find my smart phone to be a terrific tool and a significant advance over what went before. I have been contemplating this recently because I see a similarity between trying to explain *why* I am happy with my smart phone to my father-in-law and trying to explain why systemic IT improvements make sense to some of my clients.

In both cases, the audience does not share my basic premises. In both cases, there is no "killer app" and no clear analogy from what they know to what they do not know. In both cases, I am asking for a leap of faith.

For my aged father-in-law, who does not particularly want people to be able to interrupt his day with their inane chatter, the mobile phone concept never made sense. Now, when I try to explain that telephony is the minority of what I want my smart phone to do, he chortles with delighted smugness: an expensive phone that isn't even mostly a phone! Imagine the kind of technocratic, free-spending fathead one would have to be to want that! He is becoming a little uncomfortable with the ever-growing percentage of the general population who have joined me in his native city, but he is still happy to soldier on.

However, my point is not his out-of-stepness; rather my point is that my inability to explain to him what he is rejecting. He is rejecting it and then filling in the reasons later. I have failed utterly to convey that the added value is holistic and lies in the totality of the ecosystem. His eyes glaze over when I list the rather large number of functions the "phone" routinely performs for me: check bank balance? Get driving directions? Surf the web to check the hours of a museum? Share links with friends in text messages from which they can go to the same web page? Track my billable hours and expenses? Take photos of friends and family? Take photos of computer parts I need to purchase? Take notes on the train for later access on my desktop? Listen to music? Buy music? Shop for apps for work? Download apps to help me enjoy the museum in which I am standing? It is all too much and too bizarre for him.

So I feel stuck: what benefits he will allow me to present are not enough to explain the actual value of the device+apps+network yet he will not sit through the (admittedly very boring and somewhat peculiar-to-me) actual list. So we have resolved it this way: he is certain that smart phones are expensive toys and that I own one because I like expensive toys, while I shake my head in wonder at what he is missing.

He is not alone in this situation, even in my experience. I have the exact same feeling when I try to explain to IT managers why a given subsystem or middleware is worth building or improving: an endless list of small benefits just does not seem to move people who are not already intimately involved with the given business process. Like a giant jar of small change, which actually has significant value, proposals of myriad incremental improvements sit in the corner, gathering dust.

What You Want To Hear

When I entered the field of computer programming, I had some illusions about knowability, about absolute truth, about right and wrong. I thought that there were real answers to vague questions such as "which of these approaches is better?" "Who is the better coder?" "What should we do to solve this problem?"

Instead of fact-based, rational discussion I find myself often mired in religious and political muck. Specifically, I am watching a client move a data repository from a lovingly handcrafted environment to a standard environment, which standard environment blows up every time.

How did we get here? The data repository is several years old and has been a spectacular success. It was custom built for its purpose and its environment. No radical changes were indicated; so why are we watching them try to move to a dramatically different implementation?

I do not object to the idea of change; I have some sympathy for the desire to update this implementation:

• The custom environment is getting long in the tooth--almost 6 years old as of this writing--so I find it reasonable to consider replacing it.
  
• The custom environment runs Linux which is not the Unix of choice in this particular shop, so I understand their desire to port to their standard, AIX.
  
• The custom environment is based on ReiserFS which is  bit out of the mainstream, so I can see reevaluating this implementation.

What strikes me as sad is how the next implementation was chosen: the proposed implementation is what they have lying around. While the current implementation was carefully researched, painstakingly prototyped and carefully constructed for just this purpose, the proposed implementation was none of these things.

So far as I can tell, this is how the technical design decisions were made:

• The current system is old, so it must be outdated.
• The proposed system is from IBM, so it must be reliable.
• The proposed system is expensive, so it must be capable.

Conclusion: what they already bought is just the ticket. With this conclusion, everyone's judgement is validated: the techs picked the right things, the execs paid for the right things, everyone is right. Moving from the custom environment to the standard environment becomes a no-brainer: cheaper, faster, better. As I mentioned before, everyone loves an easy decision, right?

Well, there is a problem: ReiserFS is great at what it does. AIX's standard offerings have not caught up. Neither has whatever their Network Attached Storage (NAS) is running. So they can't get to step 1: copy *most* of the data to the new home. We are working on our third attempt here.

Is it a surprise that the result-oriented process yielded better results than the blind assertion of capability? No; the surprise is that anyone thought that simply declaring what was in hand to be the best solution was a good idea.

When will they admit that this proposed implementation is unworkable? I am guessing that they will never admit that and nothing we say seems to make any difference: it just isn't what they want to hear.

Smart Planning & Dumb Luck

Managing software projects requires that one make estimates about the level of effort real people will require to solve abstract problems with concrete software.

There are many boundaries in there: abstraction, concreteness, dates, level of effort, human actors and ideas. Business, and much of academia, abhors honest ignorance. Business requires staffing levels, target dates, deliverables, milestones and budgets. The most common approach I see to resolving the abstract nature of programming with the concrete nature of business is reduction-to-known: how is this project like other (preferably successful) similar projects?

In my experience, this approach is good when there is little novelty involved and is more-or-less a crap shoot when the project is highly novel. Many of our competitors handle novelty by assuming the worst and jacking up the price to cover that worst case scenario. We try to break down the project into novel and known and to do basic research so that we can turn the novel into the known as quickly and safely as possible.

But rarely does anyone, including us, talk about luck. Dumb luck. Pure chance. That sad truth is that sometimes you get lucky, sometimes you don't and sometimes you get unlucky.


(A related topic is inspiration: sometimes divine inspiration strikes, making the difficult simple and the giant task almost easy. But I prefer to think that I am divinely inspired sometimes because (spiritual version) God loves me more than He loves other programmers or (secular version) my superior neurology works this way, so I am reluctant to lump this in with luck. Although I understand that lots of other people do lump inspiration in with luck. I suspect that they are not often inspired, but that is another topic.)

I am aware of the "Luck is the residue of design" school of thought (nice context-setting here), and I mostly agree with it, but sadly sometimes I get lucky without regard to my planning or ability or anything else. Sometimes stuff just works out.

For example: sometimes I mention to a friend over a beer that I am stuck on a particular technical problem and he casually points me to just the right person to ask. Sometimes I find that I need to accomplish something and quickly and easily find just the right technical documentation on the Internet. Sometimes I find that an orphaned AC adapter fits a crucial piece of equipment. Sometimes the device driver I need only runs on an old kernel that I happen to still be running somewhere. Sometimes you get lucky: you crush the project, you beat the deadline, you come in under budget, the users are happy and the client is happy to write that final check.

Sometimes you don't get lucky: things proceed mostly as one would expect, with easy parts and hard parts and unexpected gains mostly offset by unexpected losses. These projects plod along and are delivered on time and get paid for and contribute the illusion that luck is not a concern.

Sometimes you get unlucky: critical personnel have family emergencies, supposedly supported devices don't work in *your* server, someone else's piece does not work as advertised or is late, your design is thwarted by the particular deployment environment, the client's server is plagued by gremlins, etc. These projects fray nerves, eat profit margins, sully reputations. We would all avoid them if we could.

(Beware, though, the development team that never has a bad day, or an unlucky project: it is quite possible that they pad their estimates or grossly underpromise. The only way I know to have a perfect track record is to be chronically unambitious.)

How a team or group or organization handles bad luck is very telling. The first hurdle is recognizing when you hit some bad luck and when you didn't. Very few seem able to distinguish between a failure of their team or process and bad luck.

I am not saying that blithely accepting "we were unlucky" is a good way to deal with failure. I am not defending failures of team members or of management or of process: such failures should be evaluated and corrected, either with retraining or retasking people or refining processes.

Neither do I subscribe to the position that good people never have bad luck. It would be nice if that were true, since it would greatly simplify the task of evaluating failure: there would always be blame.

I am saying that realistic appraisal is critical: neither of these simplistic approaches is a good idea.

Having determined that you got unlucky, what then?

Often the response is to avoid repeating this horrible experience by controlling variability. When practical, controlling variable in the software development process is nice, but it not a good primary goal. This is because controlling variability works best with respect to codified processes expected to produce a consistent result. It is a good way to brew beer or make steel. It is not a good way to solve problems in creative or novel ways.

Similarly, if you turn to scheduling as your protection against bad luck, you may be disappointed because if you want on-time over excellent, then you won't get excellent.

Sometimes you get unlucky. All you can do in that case, in my opinion, is pat your team on the back and let them know you feel their pain--but that you expect a return to excellence the next time.

No Substitute for User Input

Whenever I am going to deliver interactive software, I assume that there will be a significant amount of feedback and so I reserver a big chunk of time and energy to deal with that.

I was horrified recently to discover that some programmers feel that I am remiss in gathering requirements, slapdash in my implementation and lacking in my QA. I consider myself to be careful, thorough and to have a high user satisfaction rating.

When I worked through the other programmers' complaints, I discovered that their ideal software development model differed greatly from mine. (And my user satisfaction rating was much higher than theirs, from the same users no less.)

I think that it is instructive to compare and contrast the two models, so that is what I shall do.

I summarize their approach as the classic carpentry mantra, "think twice, cut once." I summarize my approach as "get feedback early and often."

Their model, which is fairly common:

1. Start with a long and thorough requirements gathering phase. The game is lost or won here.
   
2. Write up the requirements into a technical specification. Circulate that spec widely and solicit input.
   
3. Code up the spec as best you can. Write detailed user documentation.
   
4. Test the implementation. Run it through QA.
   
5. Deliver the software to the waiting users.
   
6. Handle the resulting bug reports ASAP.
   
7. Consider the job done and done well. Begin a schedule of releases and updates.

My model, which is certainly not unique to me:

1. Have a longish conversation with the users, mostly centered around them telling me what their ideal solution looks like.
   
2. Create a working prototype as quickly as possible with basic testing only and get feedback. Do a Quick Start Guide as user documentation.
   
3. Process the feedback into work to be done on the prototype. Accepting sweeping changes to the requirements or specs if those changes move you closer to the ideal solution.
   
4. As the prototype iterates through revision, it becomes the first version.
   
5. Over time, issues arise and so there is recoding through the same process.

My basic premises are these:

• Software development is a conversation between you and the users; long pauses will kill that conversation.
  
• No requirements gathering will ever get you more than 80% of the requirements, so plan for real revision at the outset.
  
• Users will happily engage with a dynamic process that produces tangible results

Upon reflection, I could see why my work looks sloppy to them: I put out many versions, I often change my design and implementation, I don't put effort into refining the coding of modules until I am sure that those modules are interested to someone. I don't tend to say "that wasn't in the spec" and I don't complain if things need to change which may seem like weakness or uncertainty to them.

On the other hand, I don't have users who complain that they wait forever for software that doesn't actually do what they need or what it to do.

As is so often the case, each side is not failing at the other's model, each side is succeeding at their own model. I am not impartial, however: I am highly confident that it is better to be one of my users.

Often In Error, Never In Doubt

This week I have a personal rant, pure and simple: I am sick of people acting as though all of their choices are perfect, without alternative and obvious.

Long ago and far away, I was introduced to rigorous decision making, a concept invented by the ancient Greeks. As part of this tradition, I was taught to carefully define my premises, apply logic to determine my alternatives and then make value judgments as to which of those alternatives I would support.

In such a methodology, there are facts, which are special because they are objective (universal), there are observations, suggestions and preferences, all of which are subjective (personal).

In attempting to make a decision, especially a decision involving a group of peers or near-peers, it is therefore important to establish the premises and agree upon the facts. Then alternatives can be generated which are both realistic and which tend to achieve the goal. Every member of the group can consider the alternatives, weigh them as they see fit, and discuss the relative merits of the alternatives and perhaps generate hybrid alternatives which have broad appeal.

Instead, I am constantly finding myself on the wrong side of this cycle:

1. There is an issue, usually not a critical one from my perspective
2. Someone senior announces a solution
3. Many someones junior voice objections
4. Objections are countered with simple denial
1. "You are wrong, that is a not a problem"
2. "Your problem exists, but it will handled by the solution"
5. The solution fails along the expected dimensions
6. Juniors are trained to pretend that the seniors were right
7. I am asked to join in the pretense while providing a non-pretend remedy

I am unclear as to why anyone would want to pretend that all problems have a clear, simple, perfect solution. This pretense is obviously absurd and continually contradicted by everyone's life experience. One of the primary skills of adulthood is the acceptance of imperfection and the pursuit of the as-good-as-possible. Why pretend that when at work, suddenly perfections are possible and always achieved by the right process?

This is particularly frustrating for someone in my position, the position of actually having to make systems work in the real world. Unlike senior management, I do not have the option of simply insisting that my work is perfect and that all criticism of it is invalid.

Now that I write that out, I can see that appeal: how awesome that would be! Except for the constantly failing part, which I would hate, even if I could browbeat people into never mentioning it.

My current working hypothesis is this:

1. We live in an age of astounding possibilities and baffling large numbers of choices
2. Our culture prizes confidence, at least in men, and rewards the confident whether or not they are correct (Often in error, never in doubt)
3. In order to be determined confident in complex, uncertain situations, one must be simplistic; keeping it simple is not enough
4. In order to be simplistic, one has to ignore complicating realities
5. Voila! we end up where I find myself: listening to falsehood asserted with authority

Of course, I could be wrong: that is part of my point, one can just about always be wrong which is why one should just about always be willing to engage dissenters and answer their objects. Alas, "Are no!" "Are too" does not constitute discussion, however often or loudly this interaction is repeated.

Mobile Meditations

We are feeling pressure to enter the mobile space. Everyone else is doing it, so we should too.

As a parent, I tell my daughter that "everyone else is doing it" is not a good enough reason to do something. But as an IT business, we find that our customers sometimes cannot evaluate us on our own merits. Instead, they need proxies to help them decide if we are good at what we do.

Proxy evaluation is sometimes hard to avoidable: I don't go into the kitchen to assess a restaurant; I look at the menu, I assess the ambiance, I rely on word-of-mouth.

So I understand the well-meaning advice that we have to enter the mobile space, to make some of our customers comfortable that we are keeping current and that we are still good at what we do. I understand that more and more people seem to rate a technology company by acquiring that company's free smart phone app and trying it out.

I have even decided to take this piece of advice and to put a mobile app or two on the docket for calendar 2012. I have also decided that discussing this decision will give insight into how we make decisions about what we do and how we do it. This will also serve as an example of an issue that is common in our practice: when something seems obvious at a high-level (create a mobile app!) but is actually quite unclear at the level of action (what kind? how fast? how expensive?)

Specifically, entering this arena raises a number of questions:

• What is the goal?
• marketing
• remove a negative (reassure customers)
• revenue
• demonstrate competence in this field
• What kind of app?
• client for a server? eg Facebook app is a UI to their server
• native app? eg native camera app
• combination? eg some games with a multi-player option
• What business model?
• free? (loss leader, eg banking app)
• freemium? (give away something, charge for more)
• for-profit? (straight up product)
• For which platforms?
• IOS only?
• Android only?
• Both IOS and Android?
• stand-alone, Wifi, 3G/4G, some combination?
• smart phone, tablet, both?
• What market?
• extend our product line to mobile?
• create a separate mobile app, perhaps an add-on to someone else?
• both?
• What development environment?
• native to target platform
• use a virtualization that runs on multiple environments

As our thought process matures, I will return to this topic to record how we decide and what we decide.

In Praise of "Throwaway" Software

A technical person who worked for one of our clients once dismissed my work as "throwaway software." I think that he meant the comment to be dismissive, but I was not insulted or perturbed; I knew exactly what he meant and I was flattered. His comment mean that I had met my goal of providing just what the users needed just when they needed it.

Like me, he had learned to program in the bad old days when development was largely unassisted and laborious. Computer time was the bottleneck, to so we conserved it. We labored over "large" chunks of code, trying to find all the bugs and typographical errors and silly mental mistakes before we dared waste precious processor time to compile our code.

When code is hard to create, hard to change and likely to be in service for years, it makes sense to move slowly and carefully, to write for the ages. "Nothing ever gets thrown away," we were told. "Anything you write is likely to be recycled and to live forever, so every line should be immaculate and immortal."

If you don't remember assembling programs into punch card stacks and then putting those stacks into wooden trays so sys admins could load them up, then you are going to have to take my word for it: coding used to be a cottage industry: we lovingly hand-crafted every line. We were thoughtful and careful and only did Important Things with the Mighty Mainframe. We were very aware of our High Priest status and we did not want to anger our computer/deity or waste our precious opportunities to interact with it.

These days, thank God, the balance has shifted dramatically: computing power and disk space are cheap, so development is less about writing fantastic code and more about refining your software so it does exactly what it needs to do. The computer helps with with syntax-aware editors, chromacoding, ease of compilation or interpretation, on-line help, etc.

With the advent of "wrong sizing," there are usually too few workers doing too much work. As mentioned before, human attention is at a premium. This includes both the humans writing the code and the humans using the code. This means that developers need to do what makes them productive in terms of writing code, and software needs to do what makes users productive at their jobs.

Often, this means writing special-purpose, "throwaway" tools to help specific users do specific jobs. If you are one of the users, struggling to do an important job, you are not going to like the description of your urgent need as "throwaway." If you need support for "only" a few months, after which the tool will never be used again, so what? You still have the need and computers can still fill it.

(To me, an "app" does a fairly high-level job in a fairly complete way, while a "tool" does a very limited job in a possibly unsophisticated or unpolished way.)

I am struck by the fact that builders don't have this hang up: special purpose scaffolding is not "throwaway building." No one says "what is the point of building concrete formers when you are just going to tear them down later?"

I watched an expert put up a shelving unit for me using a temporary scaffold, which was eye-opening. He was slight and elderly. I was bigger, stronger and younger. I offered to help him put up the unit. He was polite in his refusal of my help. I stuck around to watch because I was baffled. He eyed the unit and wall space. He threw together a scrap wood frame. He and I put the unit on the frame, at which point the unit was right where it needed to be. He screwed it to the studs in the wall with two cordless drills: the left hand drilled pilot holes and right hand used a screwdriver drill bit to sink the screws. With astonishing efficiency, the unit was up. He then removed the frame and, with the screwdriver drill running in reverse. disassembled the frame. Voila! a throwaway frame made the job almost trivial--assuming that one is skilled enough to throw together such a thing, which I am not.

I recently created such a tool to help one of our clients debug an interface. I knew going in that once the interface was working properly, the tool would be of limited use or interest. I also knew that without the tool, without the window into what was going on, the cycle of "we never got it!" being answered with "well, we sent it!" would go on forever.

As is so often the case, in the course of refining the tool and chasing down what the tool revealed, we found that there were many minor issues which appeared (falsely) to be a single, serious issue. Everyone was mostly right. No one was evil or stupid or uncaring. Once the tool gave us a shared context in which to have conversations, months of bickering were settled in days of rational, shared-reality-based email exchanges.

Is the tool "throwaway" software? Perhaps. Was it quick and easy to create? Yes. Does it provide excellent value for money? Absolutely. In this day and age, is "throwaway software" an insult? I would argue only for those living in the past.

(Note that I am not advocating writing buggy, sloppy code. Quite the contrary: weighting the effort toward refinement and debugging, in my experience, produces a lower bug count and a better user experience.)

My advice is to get out there and start whipping off solutions to small- and moderate-sized problems. Your users will thank for it, even if your software only lives as long as it has to.