Emergent Design

William Martinez as an interesting repost to Cedric Beust's recent criticism of TDD. As always William as something interesting to say, and he makes the point I made on Cedric's blog, that TDD is no the whole design picture and you need to use a different design "language" at different levels of design abstraction.

Given that William is an Architect, I seized on the idea of using "different" design languages and what I see as the over use (misuse) of modeling languages such as UML. I tried to express this in a comment on his blog, but for one reason or the other (probably due to its length :^)) his blog wouldn't except it.

Thinking it through, I believe what is central to Cedric's views on TDD is a lack of awareness of Emergent Design and how it works. It would be interesting to see Williams take on Emergent Design since it gets rid of different design roles such as Architect, Analyst and Programmer combining them into one. The way it should be :)

Getting rid of Architects? Provocative I know. Anyway I include my comment here as a description of Emergent Design. A warning. It contains plenty of strong opinions :)

Hi William,

I sort of agree. We have spent a long time trying to place "good code" in a box. You can't legislate for "good code" yet we try. The design layers you speak of are one way of stratifying code into deferent levels of design abstraction in an attempt to legislate and box up Software Development. Yet it is the code that determines the correctness of a program and ultimately the success of the design. Once we arrive at the code, all other models are merely a projection of the one true model, the code model. Other models have no concrete existence.

So how do we test the quality of our code model? Well it is like the Shrodingers Cat paradox. We do not know whether our design is correct (works) until we execute the code. We do not know whether the cat is dead or alive until we look in the box.

I think an over emphasis on modelling can lead to a situation where we forget this basic truth. Second to working code, we would also like our code to be clean. How do we know our code is clean? Well for OO systems removing code duplication through inspection works well. TDD punctuates the coding process allowing you first to check for correctness through verifiable computation (a green bar), then inspect and safely remove code smells such as duplication, coupling, poor cohesion etc with the safety net of knowing that you aren't breaking correctness.

Using this approach, your design can emerge empirically. I agree that this bottom up approach to design has its limits. It can lead to locally maximised solutions (micro-design). So how do you find the global maxima (macro-design)?

XP practices come to our aid here too. By implementing the solution one customer prioritised story at a time, you are focused to extend your macro design incrementally, the simplest way that can possibly work. So at the story level your macro design is emergent too. But that leaves one last question. Each story's implementation needs an architectural context. How do you decide on your architecture? Well other than Kent Becks "Metaphor" idea XP provides little guidance here and we are back to guessing.

Unlike traditional design though, we acknowledge that our chosen architecture is just a guess and is yet untested by our requirements (stories). Given this we make a minimal investment in our untested architecture. Some people call this a walking skeleton. As we add flesh to the bones, we may find out that the bones are in the wrong place or are the wrong size, so we change them, and in this way our architecture emerges too. We may end up with the skeleton of a Horse that emerged from the skeleton of a Pig say, all driven by our requirements and an overriding drive for simplicity.


How do you describe this process in a book? The answer is you don't. This process requires a great deal of skill and judgment applied at each punctuated phase/cycle. Top designers like Kent Beck design this way, lesser designers can learn this approach by example and through experience and practice. The idea of learning from a Teacher and developing your skills through practice is not revolutionary and is the norm for most creative professions. Unfortunately we have managed to fool ourselves into thinking that programming is not a creative profession. So we do not have the teachers and coaches, and we do not provide newbies with the concrete feedback they need to learn from their experiences. If a programmer ends up having to code around your architecture and never gets an opportunity to communicate this to you, how will you learn and improve?

It is true that not all modelling can happen at the keyboard whilst writing tests/code. But it is possible for a lot more modelling to occur this way then people think. I would say that as much as 90% of design time can be spent at or near the keyboard, and the other 10% can be ephemeral (back of a napkin, white board session, CRC cards etc). I would also say, that modelling at all levels of abstraction should be done by the same person. Remember that code is merely a human readable representation of the computational model, and it is the computational model that is our final program.

So how do you get here? Well firstly it needs to be experienced to be believed. You end up with a design that actually fits your problem, rather than a design that you thought would fit your problem, and got locked into early on. Have you ever got to the end of a project and thought if I knew at the beginning what I know now I would have designed it completely differently? Well with emergent design you get to change the design every time you learn something new. No need to guess during a BUFD (Big Up Front Design) phase.

An example of BUFD (IMO) is how we all have got locked into ORM, when it may not necessarily be the best fit for the type of problems (CRUD) we are trying to solve. I think we have become model obsessed. Steve Yegge has an interesting blog post where he explores this further. I think Steve is right.

We need to stop aping other professions and spend more time mastering our own. Software Development is a unique craft that calls for a special set of skills that cross a number of layers of abstraction. Unlike the building trade it doesn't lend itself to stratification. We need to be able to program, design, architect and analyse all at the same time in small micro cycles each lasting as little as a few seconds. We need to punctuate the process with concrete feedback. At each cycle we need to focus on a specific problem one at a time. Our cognitive abilities are finite and we must divide and conquer, so as not to overload them. And we need to allow our design to emerge, deferring design decisions to the last responsible moment, where we have learned the most and are best equipped to make the right decision. We also need to recognise when to break out in the light of new information, and reconsider the big picture, either as an individual or as a team exercise.

These skills may not fit neatly into the stratified organisational structures of our current software development organisations. But the Uber programmer doesn't fit into this world either. We need to be producing more Uber programmers who are creative, highly skilled, and massively productive. We also need to develop organisational structures where Uber Programmers can thrive. Cooperating with other programmers and other parties like the customer; all working together collaboratively as a single team with a single goal: working software. Common code ownership, and using code as a design comunication tool is aligned with this view.

We need to be creating more Kent Becks.