SCA Meme & Feedback Cycles

As memes move through a medium, they trigger new memes (diffraction) and feedback loops. Those loops diffract as well, setting off a general wave of associated tranmissions. But like ripples in a pond, feedback is exactly that, feedback, and it eventually returns to its point of origin for a resolution.

SCA (Service Component Architecture) is an SOA construct. Note its four-year rise but also note the technical analysis. Each low is a higher low, each high is a higher high. There are seven separate feedback cycles which return to their origin for resolution, where they re-affirm the rise. It's similar to tidal patterns at the beach. As the tide is rising, each new wave washes up further onto the beach.

Feedback cycles eventually return to their point of origin for resolution.
This is WHY technical analysis is predictive.

Dual Differentiation Design

I did this design for a small insurance company but all company-identifiable information is removed from the diagrams. The Dual Differentiation Model uses two abstract factories to create a matrix of differentiated functions. The first factory supplies differentiated business objects or "state". The second factory supplies differentiated services, processes which manipulate the business objects. This design has high flexibility with only two points of control. Well, three if you count the security Aspects.

Assumptions - Business objects have shared functionality of 90%+ and processes are 90%+ similiar. If differentiation for objects or process was more than 20%, I'd re-think this design. It's more complex than most of my previous designs but far more powerful. The original goal was a structure to accomodate the on-going discovery of unknown and new business rules. In retrospect, I believe the trade-off of complexity for flexibility is positive.

I used aspects for the security and logging which worked well. The aspects use the Data Access Object to enforce Read/Write/Create/Delete access on each table so the class structure isn't degraded with extraneous code and the aspect code is simple, easy to maintain. A few security exception cases (2 or 3% of requirements) didn't fit well into the aspect security model because they required selective access to groups of elements in lists, or created large result sets. The database is normalized with a 1:1 mapping of table to business object and an associated extended attributes table to support subclassing of each business object. The Business Process Model executes sequences of abstract service signatures which the Web Service Registry resolves to concrete services.

Two points of central control means a finite number of test cases. It would be easy to code recursive unit tests for all combinations of business objects against all possible business process. Okay, it wouldn't be an absolutely exhaustive test but it would probably catch over 50% of all functional bugs with little testing effort.

This application runs in a company-wide protocol adapter pattern. Incoming data is translated into a standard internal format (usually a superset of an existing industry standard) and later exported back into external formats. Essentially, the ETL layer of web services returns translated business domain objects which are used with the standard application web services.

I may code a small web-based example of this design and host it for demonstration purposes.

J2EE, SOA and P2P Cost Curves

Many organizations start with an ad-hoc methodology for their IT infrastucture, and over time it acts as a point-to-point system. If we define transaction costs as the sum of all costs over time for development, maintenance and operation, then a P2P infrastructure resembles an exponential (n-squared) curve against organization size (as a proxy for total complexity).

J2EE introduces a centralized hierarchy and standardized APIs, which impose higher startup costs, but which increase at a flatter rate for larger organizations.

Service Oriented Architecture introduces a formal message backplane which functions as a mediator, and it's probable that SOA's cost curve is flatter than J2EE. This graph shows how framework cost might be associated to organization size:

For small organizations, the cost-effective choice might be LAMP or simple programming tools.

The choice for mid-sized companies is probably J2EE.

Although SOA and J2EE aren't direct competitors, SOA has no dependencies on a J2EE infrastructure. For larger companies, it's certainly possible that alternate implementations of SOA might replace marginal J2EE installations.

Nucleus Of A Methodology, The Pain

Why my interest in SOA methodology?

Because there's money in a clear understanding of SOA implementation methodology and in the creation of Enterprise Service Bus frameworks. An effective SOA implementation for a Fortune 1000 company could run into tens of millions. A refined methodology which reduced potential failure could be worth a million when applied across many multi-million dollar projects.

Current Deductions -

1) Use a commercial ESB bus. This would reduce potential failure by a great deal. A good estimate is difficult, perhaps a 10-50% reduction in the project failure rate?

2) Pursue a minimal implementation. Complexity is multiplicative, not additive. Compare the features of a full implementation versus a minimal implementation, from the IBM SOA Redbook -

Full Implementation contains 50+ features.
Minimal Implementation contains 15 features.

That's got to be 3X to 10X greater complexity, which means the project failure rate skyrockets.

3) Use an existing security model which is ESB-specific. Again, this is an issue of exponential complexity for marginal gain.

4) SOA / ESB is an appropriate methodology for an heterogenous environment. I doubt if it's justifiable in a small scale or homogenous environment.

5) ESB's value is that it functions as a mediator. It encapsulates and isolates state information and minimizes object dependencies. This should be quantifiable in terms of transaction costs using prototype models. Total # of transaction costs could be a proxy of total project complexity, which is a proxy for statistical failure rate. There is a market for a modelling tool which predicts transaction costs. An even better tool would compare legacy transaction costs against proposed SOA implementations.

6) The complexity risk in adapter connectivity is small. Each adapter might need transaction and security legacy code, but they are independent of each other so there's no scalability or exponential complexity issues. I would tend to factor out the Adapter component of the preceding model, it has minor relevance.

7) In retrospect, the heterogeneous component of the model is probably fairly easy to predict. If we have a rough measure of total complexity (see #5), we know one of three factors in the equation. The remaining factors, team size and tool set, are driven by adoption of frameworks, project timeline and acceptable risk.

Big failure issues -

1) Managing complexity and rate of change in WS-Policy structure.
2) Clean separation of business logic from bus logic. This is a nebulous area.

----

I'm flattered, but why would the European Patent Office take such an interest in this site?

Nucleus Of A Methodology

I watched a Service-Oriented Architecture project fail first-hand. Technically it's not failed yet, but it seems likely now. I don't have the background to know the true cause, but I posted the quagmire of SOA in January of this year -

"IBM carefully crafted a system design which attempts to span the range of all things to all people. This could be a possible failure point of SOA."

I believe that SOA's greatest weakness is at the operational level. The strategic level of SOA is well-defined with training, documents and templates. My experience at coding and low-level implmentation is that tactical level is sound, too.

What's left is the operational level - a methodology and structure to convert strategic goals into tactical constructs. So let's draw up the nucleus of an operational structure for SOA -

What do we know that's concrete?

1) There's a high-level SOA structure which should take priority over tactical requirements.

2) At any point in time, there's an optimum project complexity that's achievable by the latest toolset and a certain team size. As team size grows, risk of failure rises. As complexity grows, either the toolset or team size (or both) must increase to compensate.

3) The greatest component of complexity is class differentation.

4) Existing frameworks can reduce complexity if the project is defined in a standardized form.

to be continued...

SOA Reheated

I was sceptical about Service Oriented Architecture (SOA) until last last year, in spite of this rising memegraph. I worked with IBM's proprietary technologies in the early 1990s, so I'm no easy apologist for Big Blue. But I have been impressed with their change of direction since the Dot Com crash. Websphere works. It's a solid piece of software and has been for several years. IBM maintains a fairly credible balance of "time-to-market" versus quality. They do tend to ship products prematurely, but that's the nature of the current market and their competitors behavior, and my experience has been that they make good, i.e. fix bugs, on what they release too early.

At its most strategic layer, SOA consists of three basic design patterns, each of which addresses a certain aspect of software design -

I like the simplicity of this design and I believe that it's close on the mark towards reducing total transaction costs for building and maintaining enterprise systems.

But the strongest reason to care about SOA and Business Process Modelling (BPM) is that they represent secondary s-curves of the Information Technology wave.

They address areas of software that are too fragile and cost too much to create or change. If you buy into Harry Dent's Great Boom theory, the original technology boom from 1990 to 2000 built a foundation which is a necessary component for a secondary wave of software and technology products.

Judging from the growth curves and value propositions of SOA and BPM, it seems likely to me that they are good bets for surviving the next technology shakeout, much better than staying onboard the technologies of original boom.

---

I suppose I'll go to the Sushi event on Thursday.
Fair warning to D. - don't too surprised if I show up.

Governance Services Meme

What is 'governance services'? -

"Governance provides an overarching structure to prioritize and then support the enterprise business objectives on a strategic, functional, and operational level....It defines the rules, processes, metrics, and organizational constructs needed for effective planning, decision-making, steering, and control..."

An interesting curve which has been in an upswing for over five years. It looks likely that it is a pre-inflection point graph which means it should have a few years of accelerating growth, perhaps more.

It's interesting that, although governance is nominally linked to SOA, it was in an upswing even as SOA was collapsing during the Dot Com crash.

---

And now I'll go eat dinner and watch pretty waitresses.

SOA Security Meme

The security meme says that security is a decreasing concern in SOA architecture. I'm not sure I buy it. The strength of the decentralized design is also a weakness for centralized requirements like security. The whole structure seems so porous.

Got my first login from the Pentagon today, too.

And the two Google police monitors still faithfully follow my every move, boosting my hit count. I haven't been delisted yet, so you gotta wonder...

is it half-time with a score of -

Me: 2
Malevolent Stalker: 0

?!

The Kludge Quagmire Of SOA Development Methodology

I'm about to save you a bunch of reading time.

Read through the IBM SOA design documents, and you'll find that the preferred SOA project methodology is....all of them... a witches brew of top-down, bottom-up and sometimes meet-in-the-middle rolled into one. The implication is that SOA is so different from existing systems that current methodologies don't fit well. After the third article repeated this, I started to wonder....

"What the heck are these guys talking about?"

I sat down and pondered the strengths and weaknesses of SDLC, Ad-hoc, RUP, top-down, bottom-up approaches against what I knew of SOA. Then I drew up a set of property axes for project extremes. This a rough fit, certainly not perfect, but enough to show the general concept -

Then I mapped out where SOA tends to fall along these axes -

SOA wants to encapsulate existing legacy applications, but it's a new design paradigm. It wants to be centralized for the general system through interfaces, but delegate local control of implementation. In other words, IBM carefully crafted a system design which attempts to span the range of all things to all people.

This could be a possible failure point of SOA.

However, I can see opportunity here. The lack of formal methodology implies that SOA lies more in the realm of Art than Design. For me, the axes picture give me a better idea of the knowledge requirements for building a system, but they also give me a method for determining the weighting of methodologies within the final methodology mix. For example, if I have mostly legacy code and less need for centralized approach, I would choose a methodology mix of.... oh, perhaps 80% bottom-up and 20% top-down design. For a brand-new system with little legacy code, I might choose an opposite weighting of 20% to 80%.

I suppose I should end this with some masterful concluding remarks, but I'm only debating whether I should do Karaoke Night at Rock Bottom or Fetish Night at the Vogue. The karaoke has been a pleasant surprise, but last week at the Vogue I discovered that I am Strange even in the Land of the Strange. There's something comforting about that.

SOA & J2EE, Revisited In More Detail

Undoubtedly, the J2EE enthusiasts will argue that my transaction model is unfair; that the MVC model is more flexible than I've portrayed. My goal is not to be "fair", but to determine the strengths and weaknesses of each model. SOA is still gaining strength. Why?

Updated SOA model with more detail. Notice how SOA pushes some transactions (routing and transformation) into the services layer, which is designed to be dynamically assignable, ergo it is quite scalable with good hardware.

Updated J2EE model. Notice how the controller really is... well... the controller. Of everything.

Q: What is one of the failure modes of a centralized network?
A: Saturation... the electronic equivalent of micro-management.

You can only push so many electrons through a single point source.
(that's a derivative of Kirchoff's law).

The saturation potential of MVC (closely associated with J2EE designs) is clearly shown.

It is true that J2EE and SOA are not mutual exclusive models, but... I could do an SOA system in Microsoft .Net.
Or with straight java or visual basic. SOA is a more abstract, more scalable model for large, heterogenous networks.

This is why it continues to gain mindshare (see graph at top of page).

Other thoughts... The separation of control from creation intrigues me. I believe that delegated creation is a better model for large organizations with many bodies of specialized domain knowledge. The network controls the interface, the domain specialists control how it works. I'd be very interested to see how SOA deploys into companies doing biotech, legal or accounting work.

I'm not surprised that IBM chose AbstractFactory as the foundation for the Business Directory. In my experience, it provides the maxiumum level of heterogenuity for the minimum amount of complexity, so it spans a level of design that most business applications can fit into.

Related articles -
cost curves for SOA versus J2EE
Saving J2EE, revisited

SOA & J2EE Transaction Models

note to self: The Vogue has a good alcohol:price ratio
note to others: Apparently Norway lacks a sardonic sense of humor

Last night wasn't a total loss (the Vogue was an okay place), and I worked out two simple models for SOA and J2EE systems. I used a standard MVC pattern for J2EE and assumed a heterogenous network for SOA (to require transformation transactions).

A diagram of transaction count spanning network organization -

A visual diagram is always good. In SOA, the ESB (bus) manages all control and some routing; in the MVC, the controller manages all routing, control and creation (remember our pattern model). So centralized traffic is high in MVC.

The assumption of heterogenous network is valid for large networks & companies, but you might argue that J2EE matches SOA across homogenous networks. But even then, the transaction count for the controller rises faster than for the ESB. SOA is more scalable because it pushes transactions to the periphery of the network (into the services) and puts less incremental traffic on the bus.

I see now that the issue with SOA is network latency, which might be why the phrase "coarse-grained services" shows up so often now in the design literature.

Saving J2EE, Continued

I wrote Saving J2EE about six months ago. At that time, I omitted a set of strategies in the conclusion, but my first question was "should we even try to save J2EE?"

My answer now is "probably not".

J2EE solves a set of transaction-related problems across large systems. A rehash of J2EE will probably fix many poor implementations (I did my share of those) and extract some value from the remaining marginal set of those transaction issues.

But the more I read on Service Oriented Architecture, the more I think that it solves a different set of transaction issues in large systems, for a net gain of more value. For instance, the top level structure of SOA is a simple pattern -

The Mediator is a basic transaction-cost-cutting pattern. Coupled with adapters, I suspect that total transaction cost for large organizations is very low, almost certainly lower than J2EE's model, because I can't even work out a simple J2EE comparison model without some complex pieces.

A simple mathematical scheme could do a rough transaction count for both models, and compare them against different implementation sizes. I strongly suspect that SOA will be substantially lower across a wide range of larger organization sizes.

I'm about 90% sure that "Saving J2EE" is correct; market demand for J2EE is topping out and it will be measurable in sales terms by the end of 2007.

IBM E-Business Patterns

IBM e-business SOA pattern redbook.
It's surprisingly good; a small, well-defined set of interacting patterns that are easy to learn and useful in real-life work.

Abstract Factory & Service Oriented Architecture

Abstract factory is my favorite design pattern. So the first thing I noticed about Service Oriented Architecture (SOA) is that it's a variation of abstract factory. I sketched out two designs of the same system, one as abstract factory, the other as an SOA -

Similarities include:

• central point of control


• implementations determined at run-time


• maximum class granularity with minimum complexity

But I suppose their differences are more interesting. One of the touted features of SOA is "local implementation" combined with centralized interface control. How does that differ from AbstractFactory? In SOA, the "creation" ability of the abstract factory is shifted to the service. Another difference is that a requester needs knowledge of both the abstract factory and the created factory ("Chevy factory") but in SOA, the requester is isolated from knowledge of the service.

AbstractFactory is an application pattern, SOA is enterprise.
SOA has an explicit bus, AbstractFactory doesn't.
But in general, most differences are implied, based on their context of usage, not on the patterns themselves.