Why Is Interfacing So Hard?

In theory, we provide high-end, sophisticated data analysis and reporting tools. In practice, we provide system-to-system interfaces with data analysis attached. This is because system-to-system interfacing is really hard and is usually done really badly.

We are often asked "why is this so hard?" (sometimes asked as "why were you able to do that when our people could not?" and "why did that take so long?"). So this post is all about why interfacing is hard.

Conceptually, interfacing is easy: one system ("the source") exports a datagram (a complete unit of information) and the other system ("the destination") imports the datagram. In theory, there are many "standard" formats for electronic data interchange if both systems support at least one common format, all should be well.

But all is rarely well in the interface domain. Why not?

The most common source of pain is the mismatch of data models. This sounds complicated, but it isn't and is easily explained with an example:

Consider two medical information systems which we will call A and B. System A is patient-based: the patient is the primary index and activity hangs off patients. System B, on the other hand, is visit-based: the visit is the primary index and activity hangs off visits.

System A: patient->visit->activity

System B: visit->patient->activity

The patient-centered data model is better for patient-centered analysis: it is easy to get all the data associated with a given patient for a particular time period. But the visit-centered model is better for keeping track of the current state of the system.

Regardless of which model is "better" the interfacing is complicated by the fact that the frame of reference is different. Let us work through an example.

1. System A exports a datagram which contains a visit. System A is patient-based, so the visit is exported as a small part of the patient's data.
2. System B imports the datagram
1. translates the datagram into its native System B format
2. inserts the contents of the datagram into its database

The tricky part is how to resolve ambiguity: System B is visit-based, so if there are issues with the patient demographics, who cares? We can insert the visit and worry about the patient later. But if there are ambiguities with the visit, that is fatal to System B. However, System A has the opposite worldview: so long as the patient is properly identified, the visit can ride along. So System A may have bogus visits (duplicates, unmerged visits, etc) which will choke System B, while System B may have bogus patients (duplicates, patients identified as both Bill and William, etc) which will choke System A.

When the interfacing goes awry, both vendors will be huffy: our system is fine, our export-or-import is correct, the problem must be on the other end. And they are both sort of right.

In our experience, this is where interfacing often goes off the rails: when a mismatch between the two systems would require one or other end to bend. Not only do most vendors defend their model to the death, but many vendors can only imagine their own model: any other model is "wrong". The customer gets caught in the middle.

This is where we come in. If all goes smoothly, an interface is no more than a way to convey an export from one system to be imported into another system. However, in the real world, we have often had to build intermediate systems which take in data with one model, transform that data into another model and export from the transformed model.

So if your systems "don't talk to each other" data model mismatch is a good bet as the culprit. And if your internal group or consultant or vendor quotes a larger-than-expected cost to fix the issue, there is a chance that you are not being ripped off; instead, if you are luckly, you are being forced to pay to bridge the models.