SOA and the N-tier Fallacy

My last post on Emergent design got me thinking about other common architectural patterns (in addition to ORM) which may not necessarily be the best solution for the types of problems to which they are commonly applied.

I believe N-tier architecture where N=3 is a common choice, is yet another example of Big Upfront Design locking us into a unnecessary straight-jacket even before we look closely at our problem. Now the purpose of each tier in such an architecture is to provide a service interface to subsequent tiers. The goal being that tiers are loosely coupled and hence inter-changeable. From a clients perspective the tier you will most likely want to change is the presentation tier. So a PDA client will want a different presentation tier then a desktop client, for example. But both clients would want to share everything else from the service tier down.

All of this is common wisdom and as spawned a whole architectural vernacular of its own around the term Service Orientated Architecture. So what is wrong with SOA you ask? Well nothing, other than the term is not very specific and it tells you nothing about the type of problem to which it applies. SOA is the proverbial solution looking for a problem. So lets start this discussion again from the other direction. Lets choose a concrete problem and then identify a clean working solution.


The problem: My users want to store a bunch of related data records in a database. They also want to create, retrieve, update and delete (CRUD) these records remotely through a web browser interface .

OK. They want to store related data records. Sounds like a job for a relational database to me. They then want to view these records over HTTP. Well the verbs in HTTP (POST, GET, PUT, DELETE) sort of map to CRUD operations so why not make use of this? Also my user will view data records via HTML web forms presented to the browser. So there is inherent data coupling between my web presentation and my entity/domain model. Why not except this coupling as intrinsic?

From an Emergent design perspective, the simplest thing that can possibly work is an architecture that allows you to create data record hashmaps from HTML presentation views, and persist them in a relational database. Better still from an OO perspective, it would be even better if those hashmaps could persist themselves, and perhaps do other useful house keeping like data record validation that only they should know how to do. Viola - The simplest CRUD architecture possible. A Controller to handle the HTTP verbs and map them to actions on our database. An ActiveRecord that models database records as an hashmap and is capable of persisting itself, and a View that translates the model to an HTML presentation ready to be sent back to the browser over HTTP.

Sounds very much like Ruby on Rails to me. Now how do you think DHH got there? Did he use big upfront design, or did he choose to focus on his problem afresh and apply emergent design principles? Well it should come as no surprise to find out that Rails was harvested from a real word CRUD application that evolved over time. It should also come as no surprise either to find out that 37 Signals are big advocates of TDD. In fact Rails is the only web framework I know that has TDD built in.

One last thing. The astute amongst you are probably asking what happened to our nice separation of concerns and SOA? Well if you look at my problem statement I didn't need it (YAGNI). My client was using a web browser. I've saved myself a heck of a lot of unnecessary code and shipped my release 3 months early. Its now out there making a ROI for my customer whilst those N-tier folks are still deciding how many tiers they need. Hang-on I hear you cry. So what happens when your customer does decide that he needs another client to the system:

Problem2: Not only does the system need to support CRUD data entry by people, but in addition, we have recently gone into partnership with another company who wants to access the same data records using a remote system. So we now need some way to support this too.

The world as changed so we need to change our system to suite. Unlike the BUFD guys we will only make such an investment if driven to do so by our problem. OK. We understand how to map HTTP verbs to database actions, we also know how to map our model to HTML. Given this it makes a lot of sense to re-use this infrastructure and choose a service interface that makes the most of HTTP, we settle on a RESTful approach. Our previous CRUD based controllers still apply, but now the presentation needs to be machine readable XML instead of HTML. We could put a lot of if.. else.. logic in our controllers to determine what type of presentation to create, but we decide that this would represent a lot of code duplication, so we choose to encapsulate this logic in one place which all controllers can use. This approach is so successful that we then decide to factor it out as part of our home-grown framework. Sounds pretty much like how Rails 1.2 was born.

So now we have a Restful Application that is fully SOA aware and is built on our existing technology base. The only new technology is XML, and given that XHTML is a variant of XML, we could argue that we haven't had to introduce any new technologies at all. So we have managed to introduce a Service tier when we needed it at very little cost.

Some following this discussion may come to the conclusion that DHH just got lucky. To them my answer would be that you create your own luck. By gaining a deep understanding of the problem, keeping the design clean and simple and deferring unnecessary design decisions rather than prematurely locking yourself into false assumptions; then 9 out of 10 times evolutionary paths will present themselves without you having to plan (guess) for them upfront.

In the 1 out of 10 cases where there is no low cost option, then what your problem is telling you is that your application requires diverse personalities, and whether you plan for this upfront or evolve it later, producing such an application will be expensive. The benefit with emergent design is that by applying the YAGNI principle, you will not incur this cost unless you really need to.

Emergent Design

William Martinez as an interesting repost to Cedric Beust's recent criticism of TDD. As always William as something interesting to say, and he makes the point I made on Cedric's blog, that TDD is no the whole design picture and you need to use a different design "language" at different levels of design abstraction.

Given that William is an Architect, I seized on the idea of using "different" design languages and what I see as the over use (misuse) of modeling languages such as UML. I tried to express this in a comment on his blog, but for one reason or the other (probably due to its length :^)) his blog wouldn't except it.

Thinking it through, I believe what is central to Cedric's views on TDD is a lack of awareness of Emergent Design and how it works. It would be interesting to see Williams take on Emergent Design since it gets rid of different design roles such as Architect, Analyst and Programmer combining them into one. The way it should be :)

Getting rid of Architects? Provocative I know. Anyway I include my comment here as a description of Emergent Design. A warning. It contains plenty of strong opinions :)

Hi William,

I sort of agree. We have spent a long time trying to place "good code" in a box. You can't legislate for "good code" yet we try. The design layers you speak of are one way of stratifying code into deferent levels of design abstraction in an attempt to legislate and box up Software Development. Yet it is the code that determines the correctness of a program and ultimately the success of the design. Once we arrive at the code, all other models are merely a projection of the one true model, the code model. Other models have no concrete existence.

So how do we test the quality of our code model? Well it is like the Shrodingers Cat paradox. We do not know whether our design is correct (works) until we execute the code. We do not know whether the cat is dead or alive until we look in the box.

I think an over emphasis on modelling can lead to a situation where we forget this basic truth. Second to working code, we would also like our code to be clean. How do we know our code is clean? Well for OO systems removing code duplication through inspection works well. TDD punctuates the coding process allowing you first to check for correctness through verifiable computation (a green bar), then inspect and safely remove code smells such as duplication, coupling, poor cohesion etc with the safety net of knowing that you aren't breaking correctness.

Using this approach, your design can emerge empirically. I agree that this bottom up approach to design has its limits. It can lead to locally maximised solutions (micro-design). So how do you find the global maxima (macro-design)?

XP practices come to our aid here too. By implementing the solution one customer prioritised story at a time, you are focused to extend your macro design incrementally, the simplest way that can possibly work. So at the story level your macro design is emergent too. But that leaves one last question. Each story's implementation needs an architectural context. How do you decide on your architecture? Well other than Kent Becks "Metaphor" idea XP provides little guidance here and we are back to guessing.

Unlike traditional design though, we acknowledge that our chosen architecture is just a guess and is yet untested by our requirements (stories). Given this we make a minimal investment in our untested architecture. Some people call this a walking skeleton. As we add flesh to the bones, we may find out that the bones are in the wrong place or are the wrong size, so we change them, and in this way our architecture emerges too. We may end up with the skeleton of a Horse that emerged from the skeleton of a Pig say, all driven by our requirements and an overriding drive for simplicity.


How do you describe this process in a book? The answer is you don't. This process requires a great deal of skill and judgment applied at each punctuated phase/cycle. Top designers like Kent Beck design this way, lesser designers can learn this approach by example and through experience and practice. The idea of learning from a Teacher and developing your skills through practice is not revolutionary and is the norm for most creative professions. Unfortunately we have managed to fool ourselves into thinking that programming is not a creative profession. So we do not have the teachers and coaches, and we do not provide newbies with the concrete feedback they need to learn from their experiences. If a programmer ends up having to code around your architecture and never gets an opportunity to communicate this to you, how will you learn and improve?

It is true that not all modelling can happen at the keyboard whilst writing tests/code. But it is possible for a lot more modelling to occur this way then people think. I would say that as much as 90% of design time can be spent at or near the keyboard, and the other 10% can be ephemeral (back of a napkin, white board session, CRC cards etc). I would also say, that modelling at all levels of abstraction should be done by the same person. Remember that code is merely a human readable representation of the computational model, and it is the computational model that is our final program.

So how do you get here? Well firstly it needs to be experienced to be believed. You end up with a design that actually fits your problem, rather than a design that you thought would fit your problem, and got locked into early on. Have you ever got to the end of a project and thought if I knew at the beginning what I know now I would have designed it completely differently? Well with emergent design you get to change the design every time you learn something new. No need to guess during a BUFD (Big Up Front Design) phase.

An example of BUFD (IMO) is how we all have got locked into ORM, when it may not necessarily be the best fit for the type of problems (CRUD) we are trying to solve. I think we have become model obsessed. Steve Yegge has an interesting blog post where he explores this further. I think Steve is right.

We need to stop aping other professions and spend more time mastering our own. Software Development is a unique craft that calls for a special set of skills that cross a number of layers of abstraction. Unlike the building trade it doesn't lend itself to stratification. We need to be able to program, design, architect and analyse all at the same time in small micro cycles each lasting as little as a few seconds. We need to punctuate the process with concrete feedback. At each cycle we need to focus on a specific problem one at a time. Our cognitive abilities are finite and we must divide and conquer, so as not to overload them. And we need to allow our design to emerge, deferring design decisions to the last responsible moment, where we have learned the most and are best equipped to make the right decision. We also need to recognise when to break out in the light of new information, and reconsider the big picture, either as an individual or as a team exercise.

These skills may not fit neatly into the stratified organisational structures of our current software development organisations. But the Uber programmer doesn't fit into this world either. We need to be producing more Uber programmers who are creative, highly skilled, and massively productive. We also need to develop organisational structures where Uber Programmers can thrive. Cooperating with other programmers and other parties like the customer; all working together collaboratively as a single team with a single goal: working software. Common code ownership, and using code as a design comunication tool is aligned with this view.

We need to be creating more Kent Becks.

Post Agile - In search of a new Metaphor

Agile Software Development seems to be going the same way as a number of other fads in IT, such as Object Orientation and Design by Contract. I use the word fad, because of the faddish adoption curve. Firstly there are the few original proponents with a deep understanding of the new idea, then next is the herd who lack the depth of understanding, but are eager to try the new practices, which they learn by rote, parrot fashion, and then there is the demise and ridicule as the parrot fashion, "cookie cutter" adopters falter and the original idea is discarded, baby and bath water.

One of my favorite quotes is that "the label is not the thing" from the book "Language in Thought and Action" by the psychologist S.I Hayakawa. So Agile Development is just a label, not every team that chooses to use this label is doing the "thing" the original proponents had in mind. So what to do? Should we avoid labels completely in fear that they are abused? Thats one possible answer, but labels do serve a legitimate purpose. A label is a short hand that is useful in conveying a great deal of meaning. You wouldn't want to repeat the whole Agile Manifesto every time you wanted to describe projects that share its values and principals. So the label Agile is a valid one.

The problem I think is that the Agile label conjures up an image that doesn't convey the whole thing. The image is of Gazelles leaping freely through the African Bush. Swift movement, and free spirits operating without restraint.Does this image summarise the values and principles of the Agile Manifesto? Does it summarise why people do Agile development? On one level, the answer is a partial yes. From the perspective of a programmer like Kent Beck, I suppose Agile is an apt metaphor. The freedom to hustle, move quickly and use your skills to get where you need to be fast, without impediment. But does this metaphor summarise why companies like Toyota use an Agile approach? I don't think it does. For them a more powerful metaphor is Lean. This word conjures up an image of disciplined efficiency, where everything serves a purpose and waste is eliminated. No excess baggage, no fat. But does the Lean metaphor describe the whole thing?

I wonder what would have happened if the proponents of the Agile Manifesto had decided to use the word Lean instead? I suspect that board rooms would have picked up on "the thing" more readily. As its stands Agile is often misconstrued by both Managers and Developers as a collection of technical practices, even though tools and technical practices are de-emphasised in the Agile Manifesto in favor of Leadership and sound Management.

Lets look at some of the root ideas of the Agile Movement. Lets see what Edward Deming has to say (Taken from Mary Poppendiecks Leadership presentation on InfoQ):

• Appreciate the System
• A system wide view is fundamental -never sub-optimise
  
• Manage the relationships between suppliers, producers and customers
  
• Knowledge of Variation
• Most variation is "common cause" and is inherent in the System
• Trying to eliminate this variation only makes things worse
  
• Systemic problems lie beyond the powers of the individual worker
  
• To reduce variation, provide leadership in improving the System
  
• Theory of Knowledge
• Use the Scientific Method to measure and improve Systems
  
• Psychology
  
• When it comes to people, the things that make a difference are skill, pride, expertise, confidence and cooperation.
  

To me Deming sums it up well. The noticeable addition in my mind by Deming to the "science" of Management is to recognise that traditional science plays a very small role, and an appreciation of human psychology is far more important.

Perhaps this is why Agility defies precise scientific definition. You either get it or you don't. Agility is a statement about people and how they choose to collaborate. Agile skills are human powered, and are propagated by example by Leaders who understand the importance of empowering others. When talking to people who don't get it, I think I'll be avoiding the word Agile in future. Without a shared experience to pin it to, this abused label is bound to lead to confusion. For newbies perhaps I'll try the yet untainted label "Lean" instead?

On second thoughts, I think its best to avoid labels altogether and just talk about what I do and why I do it. I'm conscious that any label will inevitably fall short, because the "thing" is all encompassing. It is "Agile", "Lean", "Adaptive", "Human Powered" and... a bunch of other stuff too! Just like human pyschology the thing is multifaceted and not easily summarised.

I guess labels are great for marketing and propoganda, but not so great at conveying non-trivial ideas.