Getting There...Eventually

Something happened recently in my consulting practice that used to drive me crazy. But thanks to experience, ennui, or both, I don't object any more.

The issue was with an interaction with a client. The situation is all-too familiar: I am creating a tool for the client and he wants to good fast & cheap, which is what client seem to think "agile" means.

As part of this speed and lower cost, he keeps choosing the minimal solution to every problem. He isn't really interested in domain expertise or 30 years experience creating these kind of apps: he knows what he needs and he knows what he wants.

Specifically, he knew that he didn't need help choosing catalogue items on which to report, even though I have had to create multiple look-up tools in the past because choosing the exact catalogue item you mean is complex and difficult. Part of the complication is that the data set contains orders from the previous ordering system, which means that one needs to know the appropriate ordering system, based on order date.

However, he knew that he did not want any bells or whistles, so version 0 was a hypertext link to the legacy web version of their catalogue. After all, he and his colleagues are familiar with the current catalogue, thank you very much.

He was able to find at least some legacy catalogue numbers, but with difficulty. Turns out he didn't know the current catalogue as well as he thought: version 1 added a link to the current web version of their catalogue.

As beta-testing went on, he complained about how hard it was to find the right items, especially if one wanted to report on dates which span the two ordering systems. I showed him one of my interactive look-up tools and he found it usable, but wanting. So version 2 is adding a revamped version of my tool the prototype.

Will we get to where I started, a heavily expert interactive look-up tool? Yes, I think that we will. Is this frustrating? No, not this time around: the client is happy because he is driving the development and is confident that he is not getting more programming than he needs or wants. I am happy because this is relatively untaxing work in a down economy; in fact, I suspect that this indirect route will end up paying me more than letting me do my thing would.

Younger me would have chafed at the lack of respect and taking of credit for my ideas and my work. Current me say, relax, keep your eyes on the prize and get your affirmation somewhere else: IT consulting is unlikely to be a good source of affirmation, at least in my experience.

Modern Modularity & Iterative Development

I recently wrote an app in about two hours: a highly tailored, very powerful tool for a clerk. And I did it as a Web CGI app using straight Perl without the benefit of a rapid application development environment or even a framework like Mason.

Hurray for me. Except, of course, I didn't really do that. It is true that I wrote the app in about 45 minutes and then spent a little over an hour in a tight user feedback / iterative development cycle, resulting in the finished product. But this is the movie version, the marketing hype, the storyline that ignores all the boring ground work that went before.

It would be more accurate to say that I spent about dozen hours over several weeks on various stages of this project: still fast, still cheap, just not spectacular (or utterly unbelievable).

From my perspective, the process went like this:

1. The client defined a problem: clean up a server's disk area by removing redundant files stored by a system of ours.
   
2. I defined the reconciliation algorithm, using md5sum to confirm that files exist in both places. Along the way we encountered some horrific special cases, but that is par for the course in the real world.
   
3. I wrote a audit program to put a random sample of these files in a web page for review; our clerk reviewed several batches of these files spread out over the entire repository. We found that the issues were highly concentrated in time windows, so we tailored our approaches to fit the eras that we found.
   
4. Based on her feedback, I reused much of the audit program to create a web-based UI to support her review and I wrote a couple of small helper utilities to automate the review process that the clerk had used. She confirmed her earlier era-based findings.
   
5. I added a database table to hold the information that the clerk was tracking on her paper pad and which she had figured out was what she needed to know. This allowed us to break down the repository into manageable chunks and to start tracking the clerk's progress more accurately.
   
6. I wrote a daemon to do prep work in the background and another one to automate as much of her review as I could. This cut the number of items requiring human review by 90% and allowed us to start automatically correcting that 90% while she worked on the ugly 10%.
   
7. Once all that was in place, I wrote the final app, a UI to review the contents of the database as set up the by daemons and to add editing functionality to support saving the results of the review. Then I added a few links to those helper apps to support the reviewing and I was done.

This is how I write good user-intensive software: a design phase which moves quickly to practical applications; we figure out the ideal process and the software trails quickly behind until we have a system that we all trust and that does the job.

Note that my utilities and previous code are highly reusable in both senses:

• I can use the code with little or no modification in different apps, which means that I only have to develop and validate once but use many times.
  
• The stand-alone web helper apps are easy to string together to enrich the environment provided by the main app. This is an homage to the Unix "string of pearl" philosophy which I find works so well at the O/S level.

In theory I should get both of these benefits from the Object Oriented programming paradigm, but in practice I don't seem to.

In this day and age, the web technology allows us to have separate-but-easily-integrated pieces and modern programming languages allow us to reuse chunks of code easily so there is no excuse for not producing polished and power apps rapidly and cheaply.

Hard vs Soft Mastery

Some years ago, as part of a discussion of computer programming styles, a colleague introduced me to the concept of "hard master" versus "soft master." Be warned: I am emphatically not making any value judgments here: I do not believe that hard masters are smarter, purer, better or more detail-oriented. I do not believe that soft masters are failing at being hard masters; I do not believe that a good hard master can do anything a soft master can do, or vice-versa.

Enough about what I don't mean: what do I mean? A little Google magic got me here, the horse's mouth: Feminism Confronts Technology by Judy Wajcman. Read the selected section for details, but the summary is this: the hard master feels he has to know all the details when using technology while the soft master knows how to get what she wants.

The use of gender-specific pronouns is not accidental: frequently hard masters are male and soft masters are female. Note that while there is a significant pro-male, anti-female bias in computer programming, this is not a bias from which I happen to suffer. I have worked with women for my entire technical career, and have found female technology interaction different but not inferior. In fact, I have found that this discipline has room for, and needs both, kinds of interaction. As a general rule, I want my bit-bashing systems software from an anti-social grumpy guy and I want my application software from an empathetic, flexible and cheerful gal.

I understand that these generalizations are not universal truths, that there are grumpy women who are antisocial and mathematical while there are cheerful men who are sensitive to how technology feels to use, and all possible permutations in between.

But I do not believe that all permutations are equally probable. I find distinct trends: men tend to define technological success and men tend to be hard masters. Women have their own technological approaches and these approaches tend to be deemed inferior by men, mostly men who cannot perform these "soft" tasks themselves.

Given this observation, I was not surprised at a recent recommendation of related reading from another colleague: an article from the Harvard Business Review that considers the assertion that women make your technical team smarter. I am sure that is true, at least for my team.

In the interests of full disclosure, I will admit to being a middle-aged man who thinks of himself as a designer, as something of a non-combatant in this fight. My attitude is that I can code anything I have to, but I am not a coder. Opinions of my coding by coders varies, but mostly I think that both hard and soft masters have reason to despair at my code. So I claim to be of neither camp: I need them both to get my designs realized, unless I end up having to code one or other kind of task myself. Which happens all too often in these recessionary times.

I find the hard versus soft dichotomy in server software versus client software. For example, configuring a web server is a heavily technical task which requires a deep understanding of networking, process management and security. By contract, creating a web page is also web-related, but has utterly different requirements, including a flair for graphic art and an understanding of usability.

I also see this division in the back end (data storage & retrieval) area as opposed to the front end (data acquisition through user interaction) area. Figuring out how to store data effectively is a job for a hard master who wants to think about file systems, database formats and the interaction of caching and performance. Figuring out how to get users to enter data quickly and accurately is about understanding how human beings use software.


From my perspective, there is trouble brewing for the hard masters. It is my observation that as abstraction rises mastery softens. When there was only simple hardware, Unix and ANSI C, someone could be a hardware expert, a firmware expert, a Unix system software expert and complete master of their own code base. Now, with server clusters and interpreted languages and database interfaces and web servers and programming environments embedded in web servers, I just don't see that as a viable or desirable goal. How do hard masters get anything done these days? They must have to restrict themselves to rather small areas of expertise.

Not only is abstraction on the rise, but I see a very real trade off between productivity and mastery which abstraction provides. Women who are happy to cruise along at the GUI-to-create-GUI level, such as Visual BASIC, kick my ass in terms of how fast and how custom they can make apps. Who cares how deeply they understand what underpins that abstraction? (Until that abstraction bites them in the butt; more of that anon.)

There has been a shift in the balance of power between apps creators (soft masters) and systems engineers (hard masters) as time has gone by. So far as I can tell, app programmers now rule: users understand what they do (create apps), economic buyers are willing to pay for what they do. Also for systems engineers, I find that users have no idea what they do, economic buyers don't relate to them or their work and organizations feel that they get the behind-the-scenes stuff for free, as part of buying apps, or smart network hardware, or whatever.

When I started out in the early 1980s, app programmers were the bottom of the ladder, while systems programmers did the real work, got paid the real money, and wrote apps whenever they felt like it. As a designer, I find that I no longer have to start by getting hard masters to buy into my design: in fact, I find that most of my gigs come from soft masters coming to me so that I can create a framework in which they can create the apps that people want. No one is interested in the lowest levels: is this MS-Access to a local database, or MS-Access as a client for a database server? Who cares?

Of course, users should care: bad infrastructure makes for a bad user experience. But users don't care about boring old infrastructure and systems engineering because most of the time they can afford not to.


All this does not bode well for the hard master. These days, I see hard mastery in demand only in those relatively rare instances when the abstraction fails or is incomplete. When you need a DLL to extend the Microsoft desktop, or a shared object to extend the Perl interpreter, you really need that. But how often does that come up? When I started out, we had five or six programmers, all male, two of whom were apps specialists. Over time, we have tended to add apps programmers, who tend to be women, and we have moved to consuming our hard mastery as a pay-as-you-go consulting commodity: we don't need full-time hard masters anymore.

This makes me a bit nervous: will the hard master be there in the future when I need him? But for now, I am busy being part of the problem: I need more soft masters and fewer hard masters I don't see that trend reversing anytime soon.

Elaborate UI, Feeble App

A pet peeve of mine is the tendency of apps to have elaborate User Interfaces (UIs) but feeble capability.

I call the UI "the front end" and the other part "the back end." Once upon a time, the back end was where all the magic happened. Where we put our effort. Now, it seems, the pendulum has swung the other way and all the work seems to go into the front ends.

(There is an analogy to our celebrity culture in there somewhere.)

Specifically, I find many apps with elaborate UIs which just don't do all the things one would expect such an app to do. Generally users just shrug: it would be great if it worked that way, they sigh, but it doesn't.

For example, I recently was debugging an interface to a medical information system. I asked the users to verify the data for a given patient on a given day. They asked if I couldn't provide encounter IDs instead. Couldn't they use the app to search for the data in question? No, they could not. Instead, they could use an elaborate GUI to laborious click on all the little icons which represented encounters, scan the cramped screen for the data, and repeat the process until they found what I was looking for. Blech.

While lacking functionality, the UI for each of these same apps is full of bells and whistles: backgrounds that change color when the control has focus, lots of different fonts, dynamically created drop-down list boxes, etc. In fact, one requires a mighty desktop just to run the UI.

My theory is that today's typical GUI is so much work to build that development teams just run out of steam before tackling the functionality.

I have noticed that many current development environments encourage this: Visual C++, for instance, has a development environment that is a GUI, is about GUIs and is not so great for doing other kinds of programming, such as bit-bashing or file manipulation.

I understand how one might arrive at this point: using X-windows and ANSI C in the 1980s was exactly the opposite: the environment did little to support GUI generation and putting GUIs together in that era was painful. So we tended to be modest in our GUI ambitions.

With this generation of development environments has come a melding of front and back ends: you write the front end and the back end gets written in bits and pieces to drive the front end. This makes the distinction between front and back end difficult to maintain, which in turn leads to back ends which are amorphous--and difficult to test (how do you run 1,000 cases?) or to use automatically (how do you call the app automatically from the background?).

In fact, there is rarely more than one front end for any app: if you don't want to use the app interactively in a graphical environment, you are usually out of luck.

The only exception I see to this trend is the plastering of a thin GUI wrapper on something mighty, such as phpMyAdmin for MySQL, or GhostView for Ghostscript. I rather like these apps, but they could not avoid what I consider to be a solid architecture: there is a clean division between the front and the back because the front was done after the fact, by different people at a different time.

This split front vs back end gives the user real flexibility: one can use a command line interface, or choose from a variety of GUIs. Running through many test cases is easy. Running automatically is easy. Why don't more development teams build apps this way?

Creating GUIs Requires Some Artistic Talent

Prologue: GUIs vs TUI
There are two main kinds of UI, following two different basic human cognitive models: recognize-and-point versus remember-and-verbalize. The underlying cognitive models follow human mental development: infants and young children recognize and point, while slightly older children up through adults acquire language and use that instead. If one is lucky enough to live that long, one may start to lose language in one's dotage and end up back at recognizing and pointing.

GUIs are all about recognize-and-point: you recognize the icon or button and you use the pointer to "shift focus" to the desired object. Text-based UIs, which we used to call simply "UIs" but which I will call here "TUIs," are all about remember-and-type: you remember the right command in this context and you type it.

GUIs should have an underlying metaphor: a physical desktop for operating systems, a physical piece of paper for word processors, a physical scroll of names for music players, etc.

GUIs are really great for two kinds of job:

1. Software which one does not use often enough to memorize the commands
2. Software which is aimed at people who do not use software well

TUIs are really great for the inverse:

1. Software which one uses so often that typing is faster than pointing
2. Software which is aimed at domain experts who want power over support

TUIs have a built-in metaphor which used to be distinctly non-metaphoric: a glass TTY terminal attached to a keyboard.

Of course, the GUI model is not purely graphical: there is a text-handling component to it because nearly every human/computer interaction, at some point, needs the possibility of text entry and text viewing.

If done well, as is the iTunes interface on an iDevice (iPod, iPad, iPhone), this text entry component is so natural that you don't think about it.

If done badly, as is the SQL window in MS-Access, this text entry component is jarring and much worse than a TUI would be.

Actual Rant
GUIs in general provide many options and lots of power. I have have a problem with most of the actual GUIs I encounter: they are hard to use. In fact, most of them suck.

While the world seems to have gone completely graphical, we also seem to have decided that terrible GUIs are just a fact of life. Dilbert feels my pain:



While graphical environment are rife with possibility and provide all kinds of pre-made widgets, they do not provide, inherently, any of the following:

• a clean, consistent layout
• a useful visual metaphor for your particular context
• guidelines for the scope or direction of the guts beneath them

If you want these properties for your GUI, you have to come up with them yourself. If you are a solid software engineer without any great innate graphic arts talent, then you need to get help with the art part and stick to the engineering. Really. We can't all be Michaelangelo, Thomas Jefferson or even Edward Tufte. Better to admit that you need help than to soldier on, creating awful GUIs.

As far as I can tell, many GUIs are designed by analogy: "let's do this the way Microsoft Office would do it, even if we are trying to identify patients who have a particular condition, and Microsoft Office is trying to automate tradition clerical work." Close enough, right? So let's say that the patients are files and their surgeries are documents and maybe their test results are also documents and soon the screen is a maze of tiny controls, all basically alike, leaving the user to remember what the current set of "documents" actually represents.

The fact that you can learn to navigate a bad metaphor doesn't change the fact that it is a bad metaphor.

And yet, the sad fact is that making all your bad GUIs look like Microsoft Word is better than making them bad in their own way. If your users expect a "File" menu bar at the top, whether or not they are dealing with "files," then finding one there is comforting. If you cannot provide a clean, consistent look-and-feel yourself, I suppose copying a popular one is better than creating a original, bad-but-novel UI.

Not all GUIs are terrible: I have watched die-hard Windows users interact with iTunes for Windows, which makes no effort to be Microsoft Office-like, and those die-hard Windows users find iTunes a delight.

So go ahead and create GUIs, if that is your job, but for God's sake do the graphical part, or get someone with graphical talent to do that part for you. Please. Your users will thank you--or at least, curse your name less.

Iterative UI Design

I am a big fan of the tradition software engineering model, which I define as going through the following steps:

1. Gather requirements
2. Produce a functional spec
3. Design a solution
4. Produce a technical spec
5. Implement
   
6. Test
7. Document
8. Deploy

I think that this model is excellent for system-to-system interfaces and other projects without an important User Interface (UI) component. However, I find that this model does not work well for User Interfaces. In fact, it tends to pit the implementers against the users, which is never a good thing.

The problem I encounter is this: large amounts of work go into the pre-design, design and implementation stages; then users get a look at the prototype, or early version, or whatever, and they want changes. In the old days, we called this phenomenon "it is just what I asked for, but not what I want."

In this scenario, the development team gets defensive: they did what they were supposed to do; they don't have resources to largely re-do the project. In fact, they don't feel that they have resources to re-do much of anything: they are planning on only fixing bugs (and probably only major bugs at that). Often there are recriminations from development to the users: "you didn't tell us that" etc. Often the users are frustrated: no one asked them these questions.

I maintain that this is predictable and avoidable by the simple expedient of assuming that the UIs are different and that they require iterations of a functional spec / implementation / user feedback cycle. Get user feedback early and often. Don't plan on a one-and-done implementation.

If you accept the fact that very few people can fully imagine what a given UI would be like to use in the real world, then you won't feel like a failure if you (or your team or your users) can't do it either.

Of course, simply drawing out the development process is inefficient and ineffective: I am not proposing a longer development schedule. Instead, I am proposing a development schedule with less design time and more user-feedback sessions.

Ideally, the UI development process is a conversation between the users and the developers, which means that the feedback / change / review cycle has to be short enough that the conversation stays alive. And someone has to moderate the changes so that the process moves forward, not around in circles. You know you are off track when you start getting feedback of the form "I think I liked it better a few iterations ago."

If you try the traditional engineering model, you are likely to find that the UI is not what the users want, but was exactly what the developers set out to create, which leads to blame for the fact that the UI either has to be redone or has to remain inadequate.

On the other hand, if you iterate properly, you are likely to find that your users are happier and your developers are less grumpy, without having to spend substantially more time on the process. Instead, you spend the time in different ways.