November 2006

SOA – Creating the Intelligent Enterprise

SOA Represents a Fundamental Change in How the Enterprise Operates
SOA represents a fundamental change in the way technology is implemented within the enterprise. In a SOA, business is the key driver and technology is secondary. BPM and Workflow drive business processes which interact with services as required. The result is a dramatic increase in business agility and flexibility. This enables the business to respond quickly to client requirements and market opportunities which are imperative in today's competitive business marketplace. In fact, SOA is technology agnostic. It really does not care what technology is used which, as I will describe later, has significant benefits for the enterprise.

Functional Differences Between Monolithic Applications and SOA
Let’s first examine why BPM is central to SOA. In traditional monolithic applications, functionality is created for each application as the business requirements dictate. For example, a sales application and an order fulfillment system both have customer information interfaces which enable the entry and updating of customer information since those requirements are essential to both systems. In fact, there are probably many common requirements to both systems so functionality is duplicated in both systems. SOA takes a completely different approach. It uses BPM tools to orchestrate the functionality of systems as required. In a pure SOA world, the customer information functionality would exist in only one service and would be called whenever needed by either the sales process or the fulfillment process as defined by a business process workflow controlled by a business process execution engine. This is a theoretical world that does not yet exist in most enterprises since most enterprises have multiple vendor or custom applications which duplicate functionality. SOA has very real benefits even in enterprises with multiple systems. In order to better understand the advantages of a SOA over monolithic applications in delivering business agility and flexibility, let’s use a simple illustrate from the kitchen.

A Simple Illustration of the Different Approaches
Imagine a home kitchen with cabinets full of ingredients for baking. The baker produces cakes and pies (desired outcome) driven by his or her interest in what to make (customer requirements) according to specific recipes for each cake or pie (a business process) with required ingredients (system functionality).

In a kitchen functioning like a monolithic application environment a separate cabinet (the monolithic application) would hold all the ingredients for each recipe. So, the cabinet for a chocolate cake has flour, sugar, yeast, chocolate, butter, eggs, milk etc. (for the sake of the illustration let’s assume perishable items are refrigerated) and the cabinet for a pie has flour, eggs, sugar, pie filling, milk, etc. In the monolithic kitchen the baker has to stock each cabinet with every item needed to create the dessert (create duplicate application functionality) to ensure the ability to make what he or she desires at that time. The baker, when making the chocolate cake, then only uses the ingredients (functionality) within the chocolate cake cabinet to make the cake. While flour and sugar exist in the pie cabinet, they can’t be used since a chocolate cake is being made. If the baker wants to make a new recipe, then he or she must stock a new cabinet with all the required ingredients.

Obviously, no one bakes like this and for good reason. In the real world of baking, ingredients are stored wherever it is most convenient and each ingredient may be used for whatever recipe is being made at the time. This is exactly the concept of Service Oriented Architecture. The business process engine (recipe) controls the amount and type of information requested or received (amount and type of a baking ingredient) and the order in which systems are accessed (recipe instructions). The result is agility and flexibility within the enterprise just as there is agility and flexibility in the home kitchen to bake whatever is desired.

This is the beauty of SOA. New business processes can be implemented quickly to respond to customer or market requirements. The fundamental difference between the two approaches is that business processes drive system functionality in the SOA world, and system functionality (until the system is modified) drives business processes in the monolithic world.

Speed, Flexibility, and Agility Drive the SOA World – A Real World Example
In the SOA world, a new business process with new business rules and different data requirements can be rolled out literally in less than an hour.

A major media content provider uses a SOA for their order fulfillment. They roll out thousands of new promotions each year. The reason the number is so high is each promotion is specific to a particular region, offering, fulfillment provider, equipment type by geographical location, etc. To do this they use a visual workflow design tool (similar to Visio) in which they create a workflow with a sequence of work nodes each of which calls specific Web services necessary for that work step, e.g. address verification service or credit card service (a process execution engine manages each step to ensure it is completed). Work nodes are dragged from a palette into a workspace in the design tool and assembled into a work process. Work nodes are reusable and customizable and new work nodes can be created to meet new requirements.

Entire existing workflows can be copied, modified, and redeployed with new changes. When the workflow is completed it is automatically deployed as a wsdl document and placed into service on a SOA application server and managed by the BPM execution engine also called a workflow engine. Rendering engines then present the workflow to various user interfaces including contact center agents, kiosks at retail stores, and their company web site. A single workflow controls the business process for all three user interfaces, meaning one workflow can control the processes of thousands of users with widely varying user interfaces.

Benefits of SOA
Business Agility and Flexibility – SOA enables the enterprise to quickly change to meet market needs or business requirements. A new business process can be designed, deployed, and in production in hours. Additionally, some SOA application servers support real-time deployment with no application downtime. A version update can occur on production servers with no interruption to the business environment. Try that with a monolithic application!

Centralization of Business Rules – Business logic and business rules are moved out of proprietary vendor applications and into a centralized non-proprietary space. Business rules and processes are changed once in a single location and automatically control multiple user interfaces. Business logic is no longer stored on proprietary systems with all the associated problems of understanding the impact of changes to each system within the enterprise when a new process is implemented. Business logic is managed centrally from one location.

Vendor Agnostic – Since a characteristic of SOA is the use of a common communication structure (typically Web services), vendor systems are no longer hard-wired into the enterprise. Vendor systems sit behind Web services integration adapters which serve as the “face” of the application. Other vendor systems don’t care what operating system, database, or vendor is behind the integration adapter. They are communicating through their Web services adapters to the business process execution engine which then manages the information based upon the defined process. This enables the enterprise to update or completely change a vendor or custom application without impacting the rest of the enterprise. CIOs and CFOs greatly appreciate this ability since they can no longer be “held hostage” by an application vendor.

Reusability – As a SOA is built, the individual services are available for reuse by the enterprise. This speeds development and reduces cost. Monolithic application code may also be reused, but the speed at which it can be reused and redeployed is exponentially slower.

Centralized Management…Global Deployment – SOA enables an enterprise to use services deployed anywhere in the world to fulfill a specific business process. True, enterprises use applications deployed globally today, but SOA and Web services are far easier to implement and use globally. Additionally, SOA enables a single console to deploy, manage and monitor all services globally. Finally, SOA enables easy deployment of services globally for purposes of load balancing and redundancy. A good example would be a highly critical credit card service. SOA enables it to be easily deployed within minutes on multiple servers in different geographical locations providing load balancing and redundancy as required.

Extension and Reusability of Legacy Systems – SOA enables integration adapters to be built for legacy systems typically using Web services. Thus, these systems are open to the enterprise for new user interfaces, integration with new systems, as well as any future technology developments such as mobile devices, etc. This capability lengthens the useful life of legacy systems resulting in lower costs to the enterprise.

SOA and Data Issues
The question of data management and how SOA can correctly integrate data between systems which are not tightly coupled has been raised. Specifically, how would a sales system and a billing system using SOA understand each other’s customer ID information? In a tightly coupled environment, both systems would use a reference database to understand the correlation of customer identities between systems. But, how would SOA handle it?

The true power and flexibility of SOA is seen in its ability to handle this issue. Three fundamental characteristics of SOA address the problem. First, SOA is a flexible environment where the problem can be solved in multiple ways with multiple technologies. Second, SOA is business process driven so the process will control how the problem is solved. Third, SOA is loosely coupled so any combination of human and machine solutions may be used.

So how does this work out in real life? Web service integration adapters would be created to serve as the integration point for each system. All required data and functionality (e.g. “GetCustomerInfo” or “CalculateSalesTax”) would be exposed through the Web services interface. A workflow would control the steps for each business process and make the appropriate requests at the appropriate time. At the moment when the sales order information is being passed to the billing system by the engine, a work node or sub flow would be initialized which would call a Web service called “CorrelateCustomerIdentity”. The purpose of this service would be to use a tool to ensure the sales system customer ID referred to the same person as the billing system customer ID.

How this would occur is completely flexible and entirely up to the enterprise. Since this comparison step is part of the workflow, the workflow engine, the business user, as well as the sales and billing system do not need to know nor do they care how it is resolved. Three specific implementation solutions come to mind (and there could be many more) to fulfill the request of the “CorrelateCustomerIdentity” service. They are:

Use the same database structure as the reference table did in the tightly coupled application architecture. Build a Web services integration adapter into the database so the “CorrelateCustomerIdentity” service can request the database to compare the information and respond with the appropriate lookup information for the billing system to use. The billing system customer ID would then be placed in the Web services XML request for the billing system to bill the customer. Thus, the billing system has the correct customer ID information to complete its requirements correctly.
Use a Master Data Management solution to control, identify, update and manage the data in both systems. Of course, it would be Web services enabled, so when the “CorrelateCustomerIdentity” service made a request, it would use its business logic (including automated and intelligent comparison tools to identify if similar customer information was in fact the same person), and then return its conclusion to the workflow engine. At this point, if the correlation is made, the business logic of the workflow could send the order request to the billing system. If the correlation is not made, then exception logic (which could take any form required) could be followed. The MDM system could also proactively manage the customer ID information in both systems, and through its business logic update the records so they are identical in all systems. A discussion of MDM is beyond the scope or intent of this document, but presents numerous methods of proactively controlling customer data while adding significant business value.
Use human interaction for comparison. This solution could be used by itself or in any combination of one or two above. While the order was being placed, the “CorrelateCustomerIdentity” service could request the customer information from the billing system and a sub flow within the workflow could compare the information from Sales and Billing and if not identical could present the information to the human conducting the transaction to verify the information. The human, whether the actual customer using a Web site or IVR to place the order, or a call center agent fielding the call from the customer could evaluate the presented information and determine the correct customer ID.

Clearly, SOA provides multiple solutions for handling data issues. Additionally, each of the above solutions could be substituted for each other and without any change to the “CorrelateCustomerIdentity” service and with no impact on the other systems in the enterprise. This is the advantage of Service Oriented Architecture.

Transition from Monolithic to SOA
In an ideal SOA every component in the enterprise would be Web services enabled all at the same time. Obviously, this will not happen nor is it advisable. Most often SOA is used in a specific project within the enterprise. Typically, this is when the business needs of the project require flexibility, adaptability, and rapid change. Eventually, the advantages of SOA become more evident and it is systematically applied as an architectural principle across the enterprise. Some guidelines which may be helpful when beginning SOA within the enterprise are:

Start with a project where the advantages of SOA and Web services are clearly understood and will easily be seen.
Start with a project where change or implementation can occur in a reasonably short period of time. Look for relatively quick successes.
Remember that SOA is business focused, not IT focused, so understand who the real stakeholders are. Once the key decision makers (CFO, CEO, CSO, et al) on the business side understand the benefits of SOA, resistance on the IT side will diminish.
Avoid spending on SOA tools that are not necessary for the initial project. It is often difficult to cost justify SOA for a project if the tools are purchased as though the entire enterprise was deploying SOA. (Don’t buy the bass boat and fish finder your first time fishing. Just throw your line in the water.)
Emphasize that SOA can actually extend the life of legacy systems and save the enterprise significant costs.
Think of SOA as evolution, not revolution. Start small and build slowly.

This document is meant only as an introduction to SOA and the methods in which SOA can handle data issues within the enterprise. Hopefully, it has addressed some of the questions related to data management within a SOA and provides a better understanding of the benefits and advantages SOA brings to the enterprise.

Addendum

So when do SOA and when do Web services make sense and when don't they? Here are some thoughts...

When SOA and Web services Don't make sense...

When two systems with expected long lifecycles interact only with each other it makes no sense to add the overhead of Web services. There is no business agility to be gained.
When an enterprise always does the same thing in the same way and rarely, if ever, makes changes to its systems or business processes.
When a new application is needed with major functionality. Most companies buy applications off the shelf today rather than try to recreate the wheel. Building a complex custom application using Web services would be as difficult if not more so than building one with traditional methods. Note: Application vendors are building applications with SOA/Web services architectures since this increases their interoperability in the enterprise and they plan to sell lots of instances of the application to be used in diverse environments. So, it makes sense to endow the application with this capability.

When SOA and Web services Do make sense…

When multiple systems must orchestrate a business process which may change frequently.
When multiple systems must orchestrate multiple business processes with different business rules and data requirements.
When an enterprise would like to avoid having their business logic stored in proprietary vendor applications.
When an enterprise would like to avoid having their business logic stored in customer interfaces between systems.
When an enterprise would like to have easy access to business and enterprise performance metrics. SOA/Web services create an environment which makes performance monitoring and business activity monitoring much simpler.
When an enterprise has multiple diverse systems using different platforms and would like to avoid the future pain of rewriting multiple integrations when it adds a new system or application to the enterprise. Businesses will often avoid pursuing a business opportunity because the cost and pain of changing everything in the enterprise is just too great.