The Uncertainties of Integration Projects

Written by Cynthia Low on December 1, 2008. Posted in Articles.

Integration is burdened by a lot of misconceptions. The uncertainties of an integration project can be deep enough to evoke hundreds of questions, specific to a company’s back-end systems. This article will focus on five common thoughts where we have seen Integration sales reps dance around one or more questions, possibly because of either shortcomings in products or a lack of knowledge on how Integration really is implemented.

Building Integrations isn’t really easy, right?
How much will I need to spend on professional services, and what particular expertise will that entail?
What is the true time to develop integrations?
What are the real costs/options of the software?
With all of the Standards out there, isn’t Integration pretty straightforward?

Mid-sized organizations may undertake either internal or external Integrations, and projects can cross over data formats and system boundaries. One good example is integrating order data from an on-line store (typically in XML or an intermediary DB back-end) to an ERP system. It’s what many think of as a singular or monolithic integration: it connects two points, crossing data formats and system boundaries.

However, that’s only part of the picture. You may also want to move the information into a CRM application as a subsequent transaction. This can be thought of as a complex (or multi-step) Integration. Some other examples of Integrations involve legacy data movement, EDI, and XML Interestingly, we are seeing an increase in spreadsheet-to-application integration, as spreadsheet formats emerge as a lower-end data-exchange standard du-jour and, as such, represent a tempting staging area for incorporating into other applications.

Why integration matters: islands need to communicate within a company.

Most mid-sized companies have IT departments that are stretched very thin, for any of several reasons:

Limited staff and resources;
Lack of knowledge and difficulty in finding impartial advice;
The cost of solutions;
Lack of time to devote to implementation and maintenance;
Short-range management perspectives;
A lack of understanding of the benefits that IT can provide, and how to measure those benefits;
Lack of formal planning or control procedures.

Without these limitations, many if not all the questions would be irrelevant. But they do exist, and lead to inevitable questions.

Q: Building integration isn’t really easy, right?

A: This is the ugly truth about integration projects. Typically, they’re tough going. There are tools that can help, but you still need to be prepared to get in there and roll up your sleeves.

There are several considerations, not the least of which are the data formats to be used (How well do you understand the data formats and processes?); the processes to define; communications methods; if you are doing an external Integration, how cooperative is your partner (well-defined specs)? What information have they provided? And, what skill sets are available in-house to achieve this understanding?

Integration does not need to be difficult if customers have an understanding of the following “whats” or variables:

Ability to work with data definitions, create definitions that can be re-used;
Easy-to-use mapping tool (graphic, drag-and-drop, connect source and target info. Avoid writing code);
Business process management, something that is often overlooked. We find that companies are conscientious about the data, but often forget about the processes, and how they can improve them or accomplish more with what they have.
Flexible way to use communications adapters. Without them, a company’s infrastructure can really be over-burdened.

So, the key to making the “how” easier is understanding what the tools can do for you. There is a direct link between the capabilities of the tools and the ease of building the integration.

Q: How much and what kind of professional services will I really need?

A: It depends on a few key factors. The first, most logically, is the size of the integration project; then, the capabilities of the tools used; and especially the in-house skill sets.

For large integration projects, “point solutions,” especially where manual programming is involved, can run from 4x to 10x the cost of the software. The sophistication or complexity of the tool, versus in-house skill sets, tends to define the need for consulting. The in-house capabilities need to be aligned with the capabilities of the tool. This may not be an obvious point, but it’s important to keep in mind. You don’t give an 8-cylinder roadster to a new driver, nor send a jet pilot to operate a back-hoe; match the tool to the skills and experience of the user.

Users can, in fact, leverage a pilot project as a mentoring stage, something we see pretty routinely. The real key, where I have seen the best ROI, is to define a small pilot-project and engage a professional services group to mentor your team. This allows a skill-set transfer, won’t consume a large amount of resources, and produces a usable Integration, rather than a throw-away.

In other words, the pilot project doesn’t go away after implementation, it actually goes into production. The concept has been proven, and the results are immediate. Because of the capabilities of contemporary Integration tools, these pilot projects only last a few weeks. They can be an effective way to measure the rate of knowledge transfer and still end up with a usable result.

Q: What is the true time to develop integrations?

A: Several factors tie in to the first question (Integrations aren’t really easy!). I’ve worked with companies where the IT folks staff the new systems inside and out. They were able to grasp the tools and run full-speed. We also have the converse situation where a smaller company has a business analyst who is savvy and is thrown into the mix as the Integration person. They have “some” knowledge of the processes, a passing familiarity with the data formats and a limited background with the notions of mapping and data conversion.

Then, we have “the rest of us.” So, this is too broad a question to answer generically. There are usually metrics available, for example, to map an EDI 4010 Purchase Order into a back-end database. Even though there are tons of DBs out there, you are still doing similar things (header/detail records etc). Professional services teams usually have a good handle on these types of metrics.

The answer lies within the capabilities of the tool and how easy it is to understand and start leveraging it. A good example is a tool that can take a flat-file document and allow you to easily “discover” the structure of the document. That would save you from having to build the data definition by hand. Web services have been designed from the ground up to aid this process by providing a schematic of the services and request/response data formats within a descriptor document called a WSDL. This self-describing nature is also inherent in XML documents, but requires a dictionary to leverage the business relationships of the data.

Understanding your business goes a long way towards developing integration. That is, besides understanding the company’s own business processes, it helps to also have an understanding of the data formats. Awareness, from the general to the specific

There are no real absolute answers, just a range of times that have been observed from projects. However your first pilot project should definitely be scoped to only take a few weeks to a month. You want to attain a completion, without burdening the user with a large knowledge-transfer upfront.

Integration tools can help accelerate the process, and a few key principles come into play:

Easy-to-use visual tools reduce learning curve
Reusable objects improve development efficiency
Integration tools can reduce complexity and therefore reduce dependency on consultants

Testing the Integration is a key area that gets lost in the time estimates, so always keep that in mind.

Q: What are the real costs/options of the software?

A: By their very nature, most sales people want the prospect to buy the whole enchilada. They want the prospect to think of integration architecture and enterprise-wide solutions together. The fact of the matter is that there are packages designed for the company to buy just what they need at the present moment, and allow them to grow into the full solution later.

Oftentimes, a sales rep. has the option to sell you less than the full-blown system, but may not let you know that until you ask. Depending on your corporate setting, you may be addressing a mandate such as an EDI standard from a large trading partner. In that case, you are looking at a tactical solution with some interest in moving into Integration at a later point.

Given the complexity of enterprise-systems today, it can also be overwhelming to think, from the start, about all the touch-points where Integration would be a benefit. Being able to buy a “slice” of a full Integration solution is a valuable way to solve your immediate need, and introduce you to the technology at a hopefully lower price-point.

Finally, be cautious of hidden costs, such as training time/ramp up.

Q: With all of the Standards out there, isn’t Integration pretty straightforward?

A: Yes, one interesting thing about standards is that there are a lot of them (see graphic, for standards relevant to Integration); and they all want you to do things their way. That’s why they call them “standards.” But, there are no “formal” integration standards, just technology standards. There are ways of thinking about Integrations, templates or patterns to follow, but no hard-and-fast rules on how to do Integration. It’s not just about connecting the dots.

Quite often, standards are simply guidelines Understanding your data and processes is a fundamental requirement when thinking about Integration.

Let’s talk about one of the real reasons why Integration exists; it’s what I call the “Standards Oxymoron”… the notion of standards regarding data. There are standards for what the data looks like, how it is transmitted, etc. There is one great thing about these standards; they are so popular that you have a lot to choose from. Every flavor of data has someone attaching a standard to it, from XML (RosettaNet, large-customer proprietary formats) to the EDI (X12 and EDIFACT). That is why we are talking about Integration; if we just had one common layout for File and Database, we wouldn’t have a lot of these issues.

Another aspect is the agility to deal with new data formats, as they emerge. Spreadsheet formatting is at the forefront of this movement. Whoever would have predicted: spreadsheets as a common data foundation for Integration?

Being equipped with these questions will better prepare a company to deal with the inevitable promises and uncertainties of an integration initiative. Considering a few of points will be especially helpful.

Start with a pilot project and mentoring
Think strategically: where is your business going?
Ask yourself, “Do I need the whole picture or just the key pieces first?”
Your existing team, with their existing skills, can be successful.

Vendors ranging from software companies to integrators to some-of-each will do their best to sell their approach. Knowing the questions will better equip you to sift through the answers.

Mark Denchy is Director of Product Management at EXTOL International. With more than 20 years’ experience in the software field, he has a wide-ranging background in application development and platform integration. He works with several clients on leveraging new technologies that enable integration.

This article originally appeared in DM Review