Why Is Interfacing So Hard?

In theory, we provide high-end, sophisticated data analysis and reporting tools. In practice, we provide system-to-system interfaces with data analysis attached. This is because system-to-system interfacing is really hard and is usually done really badly.

We are often asked "why is this so hard?" (sometimes asked as "why were you able to do that when our people could not?" and "why did that take so long?"). So this post is all about why interfacing is hard.

Conceptually, interfacing is easy: one system ("the source") exports a datagram (a complete unit of information) and the other system ("the destination") imports the datagram. In theory, there are many "standard" formats for electronic data interchange if both systems support at least one common format, all should be well.

But all is rarely well in the interface domain. Why not?

The most common source of pain is the mismatch of data models. This sounds complicated, but it isn't and is easily explained with an example:

Consider two medical information systems which we will call A and B. System A is patient-based: the patient is the primary index and activity hangs off patients. System B, on the other hand, is visit-based: the visit is the primary index and activity hangs off visits.

System A: patient->visit->activity

System B: visit->patient->activity

The patient-centered data model is better for patient-centered analysis: it is easy to get all the data associated with a given patient for a particular time period. But the visit-centered model is better for keeping track of the current state of the system.

Regardless of which model is "better" the interfacing is complicated by the fact that the frame of reference is different. Let us work through an example.

1. System A exports a datagram which contains a visit. System A is patient-based, so the visit is exported as a small part of the patient's data.
2. System B imports the datagram
1. translates the datagram into its native System B format
2. inserts the contents of the datagram into its database

The tricky part is how to resolve ambiguity: System B is visit-based, so if there are issues with the patient demographics, who cares? We can insert the visit and worry about the patient later. But if there are ambiguities with the visit, that is fatal to System B. However, System A has the opposite worldview: so long as the patient is properly identified, the visit can ride along. So System A may have bogus visits (duplicates, unmerged visits, etc) which will choke System B, while System B may have bogus patients (duplicates, patients identified as both Bill and William, etc) which will choke System A.

When the interfacing goes awry, both vendors will be huffy: our system is fine, our export-or-import is correct, the problem must be on the other end. And they are both sort of right.

In our experience, this is where interfacing often goes off the rails: when a mismatch between the two systems would require one or other end to bend. Not only do most vendors defend their model to the death, but many vendors can only imagine their own model: any other model is "wrong". The customer gets caught in the middle.

This is where we come in. If all goes smoothly, an interface is no more than a way to convey an export from one system to be imported into another system. However, in the real world, we have often had to build intermediate systems which take in data with one model, transform that data into another model and export from the transformed model.

So if your systems "don't talk to each other" data model mismatch is a good bet as the culprit. And if your internal group or consultant or vendor quotes a larger-than-expected cost to fix the issue, there is a chance that you are not being ripped off; instead, if you are luckly, you are being forced to pay to bridge the models.

Paperless, ha! or, Your Seams are Showing

One of our clients recently had to back off yet another paperless initiative. We are often asked about this very thing: why is it so hard to go paperless, even here in the early days of the 21st century? We thought that this particular instance is a good illustration of a general problem.






In this case, there are many computer systems involved. There is either no integration between these systems or rather modest integration. Here is the process in question:

1. The customer either uses a third-party app to create an order, which is printed out, or writes directly on a piece of a paper, or uses the preferred app which creates an order, transmits it electronically and prints out the paper.
   
   
   
2. The paper order is brought to an intake center. In theory, the paper is just a security blanket and the order ID brings up an electronic version of the order. The intake is done, which creates an order in the system used to process the order.
   
3. The results of intake and the electronic order arrive in the processing system. The order is processed. The resulting status is sent yet another system, the one which supports status lookup.

Sounds a bit more complicated than one would like. There are more and different systems than one might expect. But historical accident and practical considerations make the process a bit more complex than theory would dictate.

In theory, the integration across systems should be real-time and seamless, which would make the number of actual systems in play almost irrelevant, so long as they all work together. Poor theory, it never gets its way in the IT world.

So why does the paper still exists? Well, consider a real-life scenario from the very recent past:

• The customer creates an order for items I1, I2, I3, I4, I5
• The intake system does not understand I5, leaving I1, I2, I3 & I4
• The processing system does not understand I4, leaving I1, I2 & I3
• The status system proudly displays an order for I1, I2 & I3
• Irate customer calls demanding to know what happened to I4 & I5
• Baffled customer support has no idea those items ever were ordered
• The customer is very dissatisfied
• We are called in to help investigate and find item rejections in logs
• The client sees how this is happening, re-instates the paper order

So the paper copy of the order is once again riding along with the rest of the material. Now when the same thing happens, but the process can compensate.
Instead of an entry in an interface log somewhere, the intake clerk can see that item I5 is an old code, for which there is a new code and the receiving clerk can see that I4 conflicts with I3 and that the customer must have meant to order something else entirely.

In this case, the paper allows the original intent to be accessible at every stage to all the interested parties, in a form everyone can read. If the integration of the various systems was perfect, was "seamless," then there would be no job for the paper to do.

Alas, the integration is imperfect and the seams are very much in evidence. So the paper stays, for now. But pity the poor systems implementer who is supposed to fix this problem, because that job is very hard. Electronic orders arrive in the background; there is no User Interface for systems interfaces and there is no user looking when the order is being processed. So how best to flag orders with issues? Whose responsibility is it to fix the issues?

In our practice, we find that logging is good, flagging is appreciated, but documentation is king: we put the error message as comment on the order so that, at every stage, everyone can see what happened. Specifically, the users who understand the process can see the messages, instead of sys admins who rarely can do anything about business process issue.

We suspect that so long as system integration has seams, critical processes will either have paper or regular failure. So don't sell that paper company stock just yet.

Managing Up

Today's topic is ticklish: managing up.

As I understand it, this term means trying to influence one's superior's expectations and behavior.

I suppose this term should mean "treating one's superiors as though they were subordinates" but that is overly literal. However, there is a high-level truth to the exaggeration: one is taking responsibility for the attitudes and actions of one's boss(es).

When I first encountered this concept, many moon ago, I found it silly. I was quite comfortable with the idea that those below me should do as I bid and those above me should bid me do whatever it was that I was supposed to do.

Over time, I understood that there was a legitimate "managing up" need: my superiors required context and feedback if they were going to make good decisions. So I grudgingly took on the task of keeping them informed, over and above what was strictly necessary.

Alas, what I see very frequently now in IT is feast or famine: either mid-level IT professionals do nothing but "manage up," leaving their direct reports wandering in the dark, or they do no managing up, leaving their superiors with ever-more-inaccurate world views.

Worse, many newish mid-level IT folks confuse "sucking up" with "managing up." They go out of their way to confirm their bosses prejudices and to avoid correcting mistaken impressions. This turns even well-meaning, talented senior managers into blunder idiots. If you don't know what works and what does not; and if you think the issues are different than they are, you will have great difficulty choosing an effective solution to the most pressing problem.

A sure sign that managing up is not working or not being done is this: there are resources and good staff working hard to accomplish their goals, but things just don't get any better over time. In fact, they get worse, although reorganizations and new software rollouts neither improve or degrade matters.

If you are working hard with decent people but the steady and amazing march of technology is not making your job easier or better, then you may have a managing up problem. If your superiors are reasonable people, start treating them as such. If your superiors punish bearers of bad news, then managing up is out and sucking up is in. May commerce have mercy on your career.

I Don't Get No Respect

WARNING: THE POST YOU ARE ABOUT TO READ IS QUITE BITTER

I have been struck recently by the general decline in status of the techno-guru and particular decline of my status as as techno-guru.

Once upon a time, email hit the general public; for me, that was in the early-to-mid 1990s. I had starting using email about a decade before, so email was a bit old hat to me by the time it reached my mother and other trailing edge technology users. I found myself frequently asked about this new email thing by people who knew me as a technology expert. While answering naive questions is not my favourite pastime, I sighed deeply  and answered with all the patience I could muster.

(Had I known that the tsunami of users would destroy the quiet, thoughtful thing that I knew as email, I would not have been so helpful or reassuring. In those ancient times, I only received few emails and each one was welcome. I corresponded with just about anyone I cared to in the ANSI C/Unix world. I remember clearly when I started getting automated "I can't read all my email, I am only accepting email from colleagues" from the big names in the field. I shortly had to do something similar as even minor players such as myself were overwhelmed by the flood of spam.)

More recently, Facebook hit the scene. This time, the situation was exactly reversed. I found that absolutely ignorant people were comfortable lecturing me about Facebook's pitfalls (always some vague concern about privacy or wasted time) or exhorting me to enjoy Facebook's benefits (usually something about the trivia of the lives of mutual acquaintances).

Whether one is pro-Facebook or anti-Facebook is beside my point, which is this: when did technical expertise become utterly irrelevant to technical discussions? Whenever I pushed back on the lectures, I found that the lecturers were happy to dismiss my requests for more detail, for evidence, for explanation. They waved their hands elegantly: they were not computer people, after all, and we were talking about the technical attributes of a piece of software, a distributed information system.

(For a humorous take on the Facebook privacy issue, watch the satirical Onion look at Facebook: http://www.youtube.com/watch?v=cqggW08BWO0&feature=youtube_gdata_player)

Would I have been interested in their personal experience as a user or non-user of social media? Yes, I would have been. That would have been valid and interesting. But no one wanted to talk to me about their personal experience; instead, they all wanted to talk to me about either the pitfalls of the software's design and policies, or the benefits of using this technology without regard to the risks.

Sadly for me, this phenomenon is not confined to parties or family gatherings; I encounter it in the professional arena as well. I find myself being lectured on software choice and deployment by MBAs who are proud of their technical ignorance and their non-technical backgrounds. Somehow, they have convinced themselves that those of us who get our hands dirty in IT are less capable decision makers. We lack the business perspective. We are like children in a toy store, grabbing the shiny technology without regard to how much it costs or how long we will want to play with it.
I suppose that there are IT experts who can't grasp the concept of a budget. There may be IT experts who want to play with shiny new technologies just because the new technologies are new, without regard to the problems to be solved. But I am not such an IT expert and I don't think that I have ever met one.

But until I figure out how to overcome the disadvantage of years of experience and a track record of successful deployment, it won't matter: I won't be part of the conversation. Maybe I need to get an on-line MBA and lie about the last thirty years of my career.

(Why lecture an expert with only ignorance as raw material? When did this become a good idea? Do people now lecture their doctors, their dentists, their accountants, their garage mechanics? From my very limited survey, the general public does lecture experts with confidence. So perhaps this is not limited to me or to my profession. Sigh.)

Simplicity: Too Much of a Good Thing?

Like many engineering types, I strive for simplicity. I seek to avoid excess complication. I rely on analysis to tame complexity and I rely on rigour to help me manage it.

That said, I am very cranky about simplistic people and their simplistic thinking and their simplistic solutions to complex problems.

To be concise, simple solutions lack complexity while simplistic solutions suffer from being too simple, from being insufficiently complex.

In practice, I find that simplistic thinking is the result of poor analysis, of not digging deeply enough, of accepting apparently simplicity as the whole story.

I feel rather self-indulgent today, so I will put this into doggerel:

Simple is a virtue,
Simplistic is a vice.
Simple might not hurt you,
Simplistic is never nice.

So what is an example of simplistic thinking? Consider patient identification; one of our systems is used to support dealing with the public, which requires that it support patient identification. At first blush, this is simple:

• require that the patient give you their unique patient ID, probably on some kind of card you gave them;

or

• require a photo ID with name, date of birth and sex

Since the client organization already has a patient registration system, they objected to our claim that we needed to build a patient identification support module. We had to do it against their objections and then only bill for it when they saw its value.

What complicates this seemingly simple task? The actual answer is horrifically long and complicated, so I will an abbreviated reply which will still make the point:

• Women often change their names when they get married; their pre-existing paper work, orders, etc, do not magically change with them, so it is helpful to keep previous versions of names, marked as not canonical, to help ensure that the driver's license / printed report / old information mismatches are not evidence of error.
  
• Children do not have photo IDs and are often known by unofficial names. Is young Billy Doe actually Ralph W. Doe? R. William Doe? Bill Doe Jr? Is this the former Babyboy Doe? Perhaps he is the former Boytwin1 Doe--or is he the former Boytwin2 Doe? By keeping a history of these transactions we can support confirming that you have the correct child, even if the caregiver can't remember which kid had the allergy test in July, as opposed to the kid who had the allergy test in June.
  
• Alas, people make mistakes and sometimes these mistakes line up with real data. There can be two John Does, one born 4/1/1941 and one born 1/4/1961 and if you mistype and have to search, you might pick the wrong one, in which case a custom module which says "this patient does not have a relevant order, but the other one does: are you sure?" will really save on dangerous and expensive patient identification mistakes, no matter how plausible those mistakes might be.

As a vendor of custom solutions for medical information process automation (yikes! what a mouthful), I am rarely called in because a situation is simple; if the problem had a simple, obvious, excellent solution, someone would have implemented that solution. I am only called in when things are complicated, hairy, ugly or all of the above. While I do what I can to make my designs as simple as I can, some problems can only be solved with involved and complex solutions.
 
Recently this pet peeve of mine has been elevated to a business issue as
our firm more frequently encounters simplistic thinking in our customer's senior management. These managers, from their lofty perches, are issuing clean, simplistic edicts which do not match the reality in which we find ourselves.

Worse, we are losing sales to less ethical competitors who promise clean, simple solutions to murky, complex problems. These solutions don't work, of course, but by the time this is well-established, the situation is the new normal and everybody loses.

So strive for all the simplicity you can muster, but shun the simplistic with all your might.