David Chappell ( http://www.davidchappell.com/HTML_email/Opinari_No16_8_06.html ) argues that SOA may not foster the service reuse that everyone has been hoping for. I think his analysis is correct, but I think with business services we at least have a reasonable hope of achieving reuse. Here we are least dealing with the way things actually happen in the world as opposed to programmer abstractions such as objects or components. That combined with the looser coupling of services gives me some hope.

The reason why frameworks like .NET are successful is they reflect years and years of experience with programming problems. Many examples of reuse (such as file systems and compilers) are so embedded in our experience that we no longer see them for what they are.

Reuse may fail here as well for all the reasons mentioned in David Chappell's analysis. At least now I feel we are on the right track.

The Reference Model for Service Oriented Architecture defines a vocabulary for building service-oriented systems. Put together by a technical committee operating under the auspices of the OASIS standards organization, it is the result of individuals and organizations representing vendors, users, governments, consulting organizations, and academic institutions.

The Reference Model (RM) sees SOA as a means for organizing and using distributed capabilities that may be under the control of different ownership domains. The RM is not an architecture. It does not attempt to make any architecture normative. It does not try to make any standard or set of standards normative.

It does provide a common set of semantics that can be used across different implementations. This does sound rather fancy. Nonetheless, just like Moliere's bourgeois gentlemen that found out he was speaking prose all his life, many industries have been using reference models all along. They just never had to define them explicitly.

An architect for a residential dwelling knows that if they use the term door or window, the builder will understand what is meant. There are widely varied implementations of doors and windows depending, for example, if you are building a space station or an igloo. Nonetheless, everyone knows what the terms mean. Many of these terms are codified in building codes, and by standards bodies, and have evolved over the years. The software architecture community moves too quickly for such evolution; this is where standards organizations can help.

Software architectures, for sure, can have views and viewpoints , but the terms in which they are discussed have to be understood.

The core concepts that the RM discusses are service, visibility, execution context,  service description, real world effect,  interaction, and contract and policy.

I will discuss these core concepts over the next few posts.

None of this work is going on in isolation, or is it intended to denigrate other work such as the WS* specifications, or organizations such as the ISO, IEEE, IETF, the Ontolog Forum or other groups. The reference model just supplies standard definitions so that it becomes easier for each group to communicate with the others.

I would like to thank all those who helped me achieve an Microsoft MVP award for Visual Developer - Solutions Architect.

"You are so young; you stand before beginnings. I would like to beg of you,

dear friend, as well as I can, to have patience with everything that remains unsolved in your heart. Try to love the questions themselves, like locked rooms and like books written in foreign languages. Do not now look for the answers. They cannot now be given to you because you could not live them. At present you need to live the question."

• Rainer Maria Rilke, Letters to a Young Poet

I learned programming in high school from a Fortran IV manual, which was like learning how to drive a car from the owner's manual—unexciting. Later, after taking an operating systems course at MIT, I gave up entirely on programming as a profession. I did not want to spend my life doing the same thing over and over again.

What made me change my mind and become a professional programmer? In large part it was Gerald M. Weinberg's The Psychology of Computer Programming, which I first read in 1982. Weinberg not only demonstrated that programming was more than technology, it is a social activity, but he showed how the social element related to the technical. In essence, he identified and addressed the types of fundamental questions that Rilke advised the young poet Franz Kappus to study. Such as:

• “What does it mean when we say a program is good?” I learned from Weinberg that a good program is as much a matter of cultural fit as technological merit. A designer has to understand that tradeoffs are made not only among technical factors, but among technical, social and economic constraints. Too often, I have seen engineers try to build the “perfect” product while ignoring ease of use or budget constraints.

• “How do you get programmers to work together as a team?” One programmer cannot do it all. Different people have different skills, different skills are needed at different parts of the project.

• “What is leadership all about?” How do you manage change and performance? Why do many managers manipulate programmers and tread them poorly and then wonder why they get poor results?
• “How do you find good programmers?” And just what does it mean to be a good programmer? Weinberg was one of the first to point out the stupidity of aptitude testing for programmers, and the importance of understanding individual psychology in dealing with programmers.

Weinberg confirmed my own intuition that software could have an enormous impact on society, and his discussion of programming as a social activity helped explain much of the “strange behavior” I saw around me as I began working on my first programs. For example, during one of my first projects, I saw that the inability of certain people to work together had more impact on the project’s outcome than the technological issues being debated.

Weinberg examined what a naïve programmer would consider just technical topics and demonstrated how the elements of human personality and interactions between people had just as much, if not more influence over the outcome of a computer programming project than the technical issues and debates.

By understanding the importance of questions such as these, even if not every question can be answered in every situation, my value as a programmer and designer transcends whatever today’s technology du jour happens to be. I would have to say that Weinberg’s book took years off my apprenticeship, and saved me much aggravation.

To this day, I view programming primarily as a human activity, with the technical merits secondary. This does not mean you can ignore the technical merits. What makes a programmer really great is not technical genius, but an understanding of the human context of what he or she is doing. Any programmer who creates a truly revolutionary and world-changing program understands this. Others did not, or did not care to, and their contributions are hidden behind or overwhelmed by others’ accomplishments.

It is incredible that a twenty-five year old programming text containing examples illustrated with technologies that many programmers today cannot even conceive of—I recently taught a programming class where not one of the students had any idea what a punched card or paper tape was—is still a great book.

I have signed the petition at: http://www.mwdadvisors.com/resources/stop-the-madness.php . SOA does not need another buzz word. I think SOA 2.0 ranks belong even ESB on the buzz word list.

This is also an experiment. We have heard of viral marketing. Let us see if we can have viral common sense.

One of the dogmas of messaging technology is that the "truth is always on the wire." In the context of interoperability that is certainly true. The message, not the platform object model that generated the message, is all that really exists between a service provider and consumer.

Like all principles it has its limits. The statement the "truth is on the wire" only means that using an agreed upon message format is equivalent to a using a common syntax for a language such as English. It does not matter how you define the message format. XML Schema, RelaxNG, or just "ask Alice" are all equivalent. Humans are better at handling ambiguity than machines, hence English syntax can be a lot looser than a message format. Nonetheless, the point remains valid.

Syntax tells you nothing about the semantics of the message. For those of you who abhor fancy terminology, semantics means nothing more or less than the real world actions that arise from processing the message.

Just like you can misunderstand an English sentence, you can "misunderstand" a SOAP message. This misunderstanding may be a programming error, or a misunderstood or mismatched policy.

For example, I send to my bank a correctly formatted message that says transfer $1000 from my cash reserve to my checking account. If the bank transfers the money from savings to checking, that is a programming error. The "wire truth" however was not violated.

Now suppose that the bank made the correct transfer, but the bank's policy (which I did not know of at the time) was to report such transfers to a credit bureau. My altered credit score resulted in a higher interest rate on the loan I was applying for. Understanding a service's policy is as important as understanding the message format.

Truth is not on the wire, truth is the real world effect of what happens when a SOAP message is processed. Truth is semantics.

Agile based software development methodologies often remind me of the story about the person who jumps off a 100 story building, and passing the 45th floor yells out "No problems yet!"

Agile based software methods have many good ideas. Their critique of the waterfall method has great merit. The best documentation is the code itself. Document based solutions do not work. But it does no good to demolish one myth only to have it be replaced by another.

To imagine that because the attempt to completely design everything up front is futile, the idea that you can iterate every few weeks and wind up with an adequate design is often wrong. That might work for a project that is strongly user interface or end-user driven. I doubt it would work for designing an air traffic control system, or system software such as Microsoft's Windows Communication Foundation. These kinds of projects have strong lifecycle requirements about safety, security, performance, or scalability. Often they require individuals to acquire new areas of knowledge or expertise.

Barry Boehm's spiral model of software development is a much better approach.¹ The idea behind the spiral model is that at each choice point in the software development process one assess the risk that the project could fail to meet its goals. Based on that analysis the next step is to mitigate that risk. It might mean doing a prototype, refining the requirements, or doing more testing. Some of these tasks may be done concurrently. Analyzing the results of these steps might cause the development process to backtrack. In all cases, the views of all the project stakeholders (customers, developers, marketing, etc.) are considered at each analysis point.

Given this approach, the classic view (from Boehm's original paper) looks like a spiral:



Since the spiral model is a risk driven process, some circumstances might dictate an agile methodology. Other cases would require other approaches. By making risk the focus, rather than a manifesto of principles there is a higher probability of making the correct choices.

Let risk mitigation guide your development process.

1. Boehm's original paper appeared in IEEE Computer 21(5) 61-72 in 1988. In 2000 he updated the model at the "Spiral Development: Experience, Principles, and Refinements Spiral Development Workshop".

One of the benefits of service oriented systems is that they are loosely coupled.

David Orchard analyzes what loose coupling means from the perspective of the Web services stack. A human being can recognize that a field in a form is misplaced, software cannot. So for a particular message invocation, early binding is necessary. This is certainly true for standards. There needs to be a defined place for addresses and security tokens.

Orchard asks us to imagine Purchase Order system. A particular piece of information in a particular message must be bound to the appropriate programming types. If you need to know the name of the purchaser, you must early bind to the format of that name. Or to use fancy language, the service must understand its semantics. But it is only necessary for those programming types that the service needs to understand. Here is where building service interactions as messages rather than as remote procedure calls (RPC) is important.

If a service interaction is defined in terms of RPC, then if you change the semantics, you must change the service interface. As long as one type of the method call changes, the whole interface is broken. If you send messages (concretely XML messages), so long as the service can find the information it needs, the service is not bound to a particular message format. Other information can change, but the service does not care.

For example, if a service processing a message does not care about security, they can ignore the WS-Security SOAP headers. Those headers can change and the service can ignore all the security possibilities. The inventory service does not care if the credit information changes.

True, if XPath is used you are dependent on a certain structure to find information, but if you mark your documents with its version, or associated XML Schema, you could use the appropriate location path for the document. Or if you want to bind everything to type you can use the appropriate XML Schema instance to serialize the message to the appropriate programming types.

Loose coupling at the application level is about inserting levels of indirection to handle versioning (so what else is new?). But a message can do this because at the service interface the message is opaque. A RPC is not opaque.

At the application level loose coupling is how easy is to make a change that does not impact other parts of the system. With opaque messaging, a new version can be added without impacting other clients. If a service wants to reject a version it no longer supports, or does not yet support, it can do so without impacting other clients. In this restricted, but vitally important sense, semantic meaning in a Web service can be late bound.

Sarbanes-Oxley mandates that public companies should be able to produce all materially relevant transactions during an audit.

In the world of service oriented architecture, huge volumes of business documents flow freely as messages between services. These services are orchestrated (or choreographed if you wish) to produced business processes. To give you some idea of the volume, some people fear that the volume of XML is starting to take larger and larger fractions of network bandwidth. This is why some are starting to push the use of Binary XML for SOA messages.

In this world of huge stores of electronic messages and documents, how in the world do you find all the relevant ones? This is where XML Schema comes to the rescue. Your XML documents should be defined with schema, and hence subject to validation. Performance considerations may dictate that you do not validate your documents during message processing. Nonetheless, with schema definitions you should be able to query your messages to search and find the relevant documents.

For example, if you need to find all transactions with a given company worth over a certain threshold, you have to the tools to find it.

Microsoft's Indigo platform will unify all the divergent transport technologies (ASMX, WSE, COM+, MSMQ, Remoting) that are in use today. For building a service on the .NET platform this is the technology you will use.

What technology should you use today?

The ASMX platform's programming model is the same as Indigo's. Attributes, indicating what technologies (security, reliability, etc.) you want the infrastructure to use are applied to methods. Hence, a converter will be provided to convert ASMX code to Indigo code.

Does this mean ASMX should be the technology of choice? I would argue that WSE is the better technology to use. WSE's programming model is not that of Indigo. Classes and inheritance are used to interact with the WSE infrastructure. WSE will interoperate with Indigo. Nonetheless, the conceptual model of WSE is identical to that of Indigo.

ASMX is tied to the HTTP transport and its request / response protocol. It encourages programmers to think of a service call as a remote procedure call with programming types, not as an interoperable, versioned XML document message validated by XML Schema.

Service developers need to think of request / response as one of several possible message exchange patterns (MEP). The most fundamental MEP, the one all MEPs are built from, as the WS-Addressing spec makes clear, is the one-way asynchronous message. Business services tend to be asynchronous; you apply for a loan and you do not hear back for days.

Service messages can go through intermediaries before reaching the ultimate recipient. Each message segment may go over transports other than HTTP.

WSE's transport classes allow you to build services that use different MEPs over various transports. The SOAP envelope classes make it easy to build the SOAP message body as XML, or serialized XML objects. You learn to think in terms of XML documents and messages, not execution environment dependent types.

Using this conceptual model your services will last longer, and be easier to evolve in a business environment. That will be of more use to your business than using a technology that has a better upgrade path, but will have to be rewritten sooner because it is poorly designed and implemented.

Grady Booch has fired another attacking missile in the great debate over software factories, and the idea's defenders have replied. Microsoft's view of the world is outlined in a series of articles by Jack Greenfield on the MSDN site:
http://msdn.microsoft.com/architecture/overview/softwarefactories/default.aspx?pull=/library/en-us/dnmaj/html/aj3softfac.asp http://msdn.microsoft.com/architecture/overview/softwarefactories/default.aspx?pull=/library/en-us/dnbda/html/softwarefactwo.asp http://msdn.microsoft.com/architecture/overview/softwarefactories/default.aspx?pull=/library/en-us/dnbda/html/softfact3.asp

The basic idea behind software factories is to move the production of software from a craft to an industry resembling manufacturing. This is not a new idea. In the second article, after reviewing why these efforts have failed in the past, Greenfield says:

"We are unable to achieve commercially significant levels of reuse beyond platform technology. The primary cause of this problem is that we develop most software products as individuals in isolation from other software products. We treat every software product as unique, although most are more similar to others than they are different from them. Return on investment in software development would be far higher if multiple versions or multiple products were taken into account during software product planning. Consequently, we rarely make commercially significant investments in identifying, harvesting, packaging and distributing reusable assets. The reuse that does occur is ad hoc, rather than systematic. Given the low probably that a component can be reused in a context other than the one for which it was designed, ad hoc reuse is almost an oxymoron. Reuse rarely occurs unless it is explicitly planned in advance."

I agree as far as he goes, but I do not think he fully comes to grips with why the idea of software factories has a long way to go. No doubt part of the reason is that the idea of a software factory resembles the idea of artificial intelligence. Every success redefines the goal. Nonetheless, I think there is a more fundamental reason.

Programming has always been a labor intensive activity. As a result, from the very start people have tried to figure out how to automate as much of the process as possible. Compilers were one of the very first attempts to automate software development. Today we take them for granted, but well into the 1970s people were still arguing over whether a good human assembly language programmer could code better than a compiler. Debuggers, linkers, loaders, file systems, operating systems, distributed transaction coordinators, were all invented to automate parts of the software development. How long did it take for the idea of a virtual machine (as in Java or .NET) to become practical for most software development?

You can, as Greenfield does, view these artifacts as improved abstractions. Abstractions are very critical to software development. I view these developments differently. I see them as automating what we understand how to automate. Code libraries such as for .NET and Java are in the same category. After years of experience we now understand enough of some of the critical elements of certain parts of software development to encapsulate them in libraries.

But software is not like other engineering pursuits such as bridge building. Most bridges, although they look different, are really one a few basic kinds. Because you cannot copy a bridge like you can copy a program, you need to build a new bridge at every place you need to cross a river. Hence you can much more easily replicate what you did before, or learn from experience.

This really came became clear to me when I was a graduate student in nuclear engineering. In the reactor design course final exam we were asked to design a cooling system for a nuclear reactor. We were not, however, to use our fundamental understanding of physics and engineering to do this. We were to apply the American Society of Mechanical Engineering (AMSE) standards for cooling systems to do the design.

Why has this been so difficult to do with software? Since software is easy to copy, you only need to create a new piece of software when you need to do something new. Automation is about understanding what you have done in the past. You cannot automate what you do not understand. So much of software development is done for things we have not done before. We do not know, or do not fully understand the domain models that Greenfield relies on for the idea of software factories. This is why the CASE tools, or the code generators of the past have not provided any reduction in cost and time. This is why I think UML based code generators will not be wildly successful either.

Of course you need to understand the domain models. It is just that in a dynamic economic environment, you do not arrive at the knowledge in time to automate the process until it becomes yesterday's understanding. As yesterday's understanding it will take a while to see if it is fundamental enough to be worth automating.

So long as software is about innovation, or doing what we do not yet understand how to do, it will always have a large craft component. Maybe automation will decrease the need for programmers, and thus reduce the labor cost. So far this has not happened in 60 years. People may use cheaper programmers, but that is another story for another time.

David Chappell, in his latest newsletter , argues that Service Oriented Architecture (SOA) promotes software reuse far better than objects do. For him, object reuse usually fails for two reasons. First, in an evolving business environment it is difficult to come up with a good definition of a business object such as a customer. Second, software developers seem to catch the not-invented-here plague fairly easily.

I certainly agree with this.

One of my favorite questions when discussing object oriented design with students is to ask for the definition of a chair. Invariably, the answers will include legs. Then I ask them if an ottoman or a bean bag chair fits their definition. Coming up with good class definitions is hard. I have been involved in software development long enough to have frequently seen the not-invented-here syndrome.

He also argues that the best examples of reuse occur with applications such as PeopleSoft or SAP. SOA reuse resembles application reuse. I certainly agree with this as well.

But, in my opinion, the fundamental reason that reuse with SOA will be more frequent than with objects is that SOA reuse is loosely-coupled black box reuse, most object reuse is tightly-coupled white box reuse. SOA is a design pattern that has benefited from our struggles with object-oriented approaches.

When you treat a component as a black box you interact with it through its external properties, or as we say in the programming world, its interfaces. You do not have to understand how a fork is built or what it is composed of. If you do not like your two-pronged desert fork, you get a three-pronged fork to eat your peas. You do not try to modify the two-pronged fork. The real world is loosely coupled.

With inheritance you must understand the object's implementation. You must always be aware of the fragile base class problem. You can change the behavior of existing programs if you are not careful. This only reinforces the not-invented-here syndrome because it is perceived to be easier to write classes from scratch, than to try to understand another programmer's code. Objects are tightly coupled because they are connected through a stack and a linker/loader.

Modern object oriented practitioners have realized this. The design patterns community emphasizes composition/delegation over inheritance, and interfaces over implementation. You use the component through its interface treating it as a black box. Interfaces produce looser coupling.

SOA is a design pattern for building business process in the real world. It does not tell you how to architect your application. It does not provide an implementation. It does tell you to construct your services completely independent of each other. Services are independently deployed. Code or database tables are not shared between services. Services in a SOA interact through a loosely coupled interface that is defined as a series of messages.

SOA is a continuation in the large of what good object oriented designs have started to become: less emphasis on the classes, and more emphasis on the loose coupling provided by interfaces. Obviously objects provide the actual implementation, but that is not how the users of those implementations view them. They view them as black box interfaces.

Class frameworks such as J2EE and .NET have been successful. Both those frameworks, encapsulate stable, well-understood problems that the software world has been working on for over 50 years. Not only are the problems of business not well understood, the environment is quite dynamic. Services in a SOA are loosely coupled black boxes because they reflect the loosely coupled real world.

Everybody talks about how the New England Patriots Super Bowl win last year was a team effort. Whether it was the backup quarterback imitating Peyton Manning on the scout team, the statisticians, the players, the coach, or the personnel guy, everybody contributed.

Teamwork, of course, is one the perennial topics du jour in the software world. Demarco and Lister’s classic Peopleware talks about it, introducing the concept of a “jelled” team. All the variants of Extreme Programming rave about it.

But what makes a team good? It is difficult to have a good discussion about software teams because there are not enough public concrete case studies. Could sports teams provide a basis for such a discussion?

Peter Drucker, the management expert, thought so. In a Wall Street Journal essay written several years ago, he discussed three paradigmatic teams: baseball, football and tennis.

What type of team does your organization have? As Drucker makes clear in his essay these teams are distinct alternatives. They have unique strengths and weaknesses, but attempts to combine parts of each are a recipe for disaster.

I bet a lot of traditional software organizations have baseball style teams. You do not play as baseball team, you are a member of the team. The third baseman never pitches, the tester does not do development. Designers do not have much interaction with developers. This is the old-style Detroit assembly line. The big advantage to this team is that it makes it easy to train and evaluate personnel. Everybody can be a "star" no matter how difficult they are to get along with. On many plays, certain players are not important. The left fielder does not do much on a routine ground ball to the second baseman. Symphony orchestras are like this as well.

This approach works well when the task is well understood and can be reduced to "routine". Here one can understand the drive to outsource and offshore software tasks. If things are well defined, and competence exists elsewhere, then price drives all.

The problem comes when you need to innovate quickly.

Football teams are more flexible. There is no equivalent to the halfback option pass in baseball. On almost every play, every player is necessary, if for no other reason then to prevent some other player from getting to the ball carrier, or the quarterback. Everybody works in parallel. Unlike baseball, everybody has to follow the coach's orders or else the team will not win. You have to train together to be effective.

How do you evaluate and train people? An individual's value is often related to how they complement the rest of the team, not only on their individual strengths and weaknesses. Play without a linebacker or a safety, and you will fail just as if you played without a quarterback. But why do quarterbacks get more money? Why do they both get more money than teachers? This is the old diamond-water paradox in economics ( http://www.libertyhaven.com/noneoftheabove/sports/praiseathletes.shtml or http://www.ace.uiuc.edu/Classes/ace100/lectures/Lecture_1.pdf ). Here you have to reward people based on their marginal value to your team, not on some absolute scale. It is much harder to outsource, much less offshore if you live in this world.

You still need, as Drucker points out, a score to evaluate how well the team is doing. Though as any football fan will tell you, the score does not relate well to how an individual player is doing. That is why you have to "watch film" to evaluate players, something baseball sabermetricians ( http://www-math.bgsu.edu/~albert/papers/saber.html ) do not have to do. Software managers in this world really have to understand what is going on in order to evaluate and train their people.

Finally there are tennis doubles teams (or jazz combos). There are no clear players, only roles to be filled by different team members at different times. This is the ultimate in flexibility and adoptability to changing circumstances. But there really has to be a fit here. How do you train and compensate in such a world where you can succeed or fail, but there is no score. It is hard to relate the end result to the individual. You certainly cannot outsource or offshore here.

As Drucker says, teams are tools, and you have to understand your environment and pick the appropriate approach. This is what management is all about.

One of the recently discovered Internet Explorer bugs allows malicious sites to install key stroke recording code on your system. This certainly has got a lot of press and deservedly so because of the widespread presence of IE as a browser.

Every time this happens I wonder, does open source produce more secure code? Do “more eyeballs” reviewing the code produce better code? Looking over the list of vulnerabilities on the US-Cert Issues Advisory list makes me doubt that this is true.

Based on my experience, too many reviewers often make for poorer reviews. Remember the last time you had to sign off on a document with a long list of reviewers? The early reviewers glance at the document knowing more reviewers will look at it later. The later reviewers assume the early reviewers did most of the work already. The result is a lackadaisical, poor job of reviewing. You cannot tell me that the open source community is immune from the natural tendencies of human nature.

Approval by committee is no different than design by committee. Just because the committee is larger does not automatically make the review better.

Driving on the highway around Boston I was wondering about its virtual counterpart, the Information Superhighway. Massachusetts’s accident rate is the highest in the country. People mutter in frustration, “You can’t get there from here” as they navigate streets that look as if cow meanderings determined their path. Yet people and commerce move with an ease and openness that can only imagined on the Information Superhighway.

What makes one so open and the other not? Some might proclaim “Heed the Three Opens of Modern Enterprise Architecture: Open Source, Open Standards, and Open Data.” In my mind I compared the concrete and virtual parallels.

The most obvious analogy is Open Standards. Traffic laws allow for vehicles to travel. Vehicles must be able to signal turns. Vehicles have to stay in lane. They have stop lights and backup lights. In fact any vehicle that follows these standards is allowed on the road. Much to the chagrin of many drivers, following these standards allows bicycles on the road. Standards can even allow for varying defaults. Everywhere but New York City has “right turn on red” as the default. When vehicles arrive at their destination then the work begins.

Vehicles are similar to messages. Open standards define the contents of a message and allow them to get to their proper destination. When the messages arrive at their destination, the actual work begins. Here Web Services standards (SOAP, WSDL, WS-Security, etc.) seem to have reached critical mass. While much more work needs to be done, the industry seems to understand what must be done (routing, federated security, etc.), although in some areas, such as transactions, it is not clear w