Thursday, March 27, 2008

Emergent Design

William Martinez as an interesting repost to Cedric Beust's recent criticism of TDD. As always William as something interesting to say, and he makes the point I made on Cedric's blog, that TDD is no the whole design picture and you need to use a different design "language" at different levels of design abstraction.

Given that William is an Architect, I seized on the idea of using "different" design languages and what I see as the over use (misuse) of modeling languages such as UML. I tried to express this in a comment on his blog, but for one reason or the other (probably due to its length :^)) his blog wouldn't except it.

Thinking it through, I believe what is central to Cedric's views on TDD is a lack of awareness of Emergent Design and how it works. It would be interesting to see Williams take on Emergent Design since it gets rid of different design roles such as Architect, Analyst and Programmer combining them into one. The way it should be :)

Getting rid of Architects? Provocative I know. Anyway I include my comment here as a description of Emergent Design. A warning. It contains plenty of strong opinions :)

Hi William,

I sort of agree. We have spent a long time trying to place "good code" in a box. You can't legislate for "good code" yet we try. The design layers you speak of are one way of stratifying code into deferent levels of design abstraction in an attempt to legislate and box up Software Development. Yet it is the code that determines the correctness of a program and ultimately the success of the design. Once we arrive at the code, all other models are merely a projection of the one true model, the code model. Other models have no concrete existence.

So how do we test the quality of our code model? Well it is like the Shrodingers Cat paradox. We do not know whether our design is correct (works) until we execute the code. We do not know whether the cat is dead or alive until we look in the box.

I think an over emphasis on modelling can lead to a situation where we forget this basic truth. Second to working code, we would also like our code to be clean. How do we know our code is clean? Well for OO systems removing code duplication through inspection works well. TDD punctuates the coding process allowing you first to check for correctness through verifiable computation (a green bar), then inspect and safely remove code smells such as duplication, coupling, poor cohesion etc with the safety net of knowing that you aren't breaking correctness.

Using this approach, your design can emerge empirically. I agree that this bottom up approach to design has its limits. It can lead to locally maximised solutions (micro-design). So how do you find the global maxima (macro-design)?

XP practices come to our aid here too. By implementing the solution one customer prioritised story at a time, you are focused to extend your macro design incrementally, the simplest way that can possibly work. So at the story level your macro design is emergent too. But that leaves one last question. Each story's implementation needs an architectural context. How do you decide on your architecture? Well other than Kent Becks "Metaphor" idea XP provides little guidance here and we are back to guessing.

Unlike traditional design though, we acknowledge that our chosen architecture is just a guess and is yet untested by our requirements (stories). Given this we make a minimal investment in our untested architecture. Some people call this a walking skeleton. As we add flesh to the bones, we may find out that the bones are in the wrong place or are the wrong size, so we change them, and in this way our architecture emerges too. We may end up with the skeleton of a Horse that emerged from the skeleton of a Pig say, all driven by our requirements and an overriding drive for simplicity.

How do you describe this process in a book? The answer is you don't. This process requires a great deal of skill and judgment applied at each punctuated phase/cycle. Top designers like Kent Beck design this way, lesser designers can learn this approach by example and through experience and practice. The idea of learning from a Teacher and developing your skills through practice is not revolutionary and is the norm for most creative professions. Unfortunately we have managed to fool ourselves into thinking that programming is not a creative profession. So we do not have the teachers and coaches, and we do not provide newbies with the concrete feedback they need to learn from their experiences. If a programmer ends up having to code around your architecture and never gets an opportunity to communicate this to you, how will you learn and improve?

It is true that not all modelling can happen at the keyboard whilst writing tests/code. But it is possible for a lot more modelling to occur this way then people think. I would say that as much as 90% of design time can be spent at or near the keyboard, and the other 10% can be ephemeral (back of a napkin, white board session, CRC cards etc). I would also say, that modelling at all levels of abstraction should be done by the same person. Remember that code is merely a human readable representation of the computational model, and it is the computational model that is our final program.

So how do you get here? Well firstly it needs to be experienced to be believed. You end up with a design that actually fits your problem, rather than a design that you thought would fit your problem, and got locked into early on. Have you ever got to the end of a project and thought if I knew at the beginning what I know now I would have designed it completely differently? Well with emergent design you get to change the design every time you learn something new. No need to guess during a BUFD (Big Up Front Design) phase.

An example of BUFD (IMO) is how we all have got locked into ORM, when it may not necessarily be the best fit for the type of problems (CRUD) we are trying to solve. I think we have become model obsessed. Steve Yegge has an interesting blog post where he explores this further. I think Steve is right.

We need to stop aping other professions and spend more time mastering our own. Software Development is a unique craft that calls for a special set of skills that cross a number of layers of abstraction. Unlike the building trade it doesn't lend itself to stratification. We need to be able to program, design, architect and analyse all at the same time in small micro cycles each lasting as little as a few seconds. We need to punctuate the process with concrete feedback. At each cycle we need to focus on a specific problem one at a time. Our cognitive abilities are finite and we must divide and conquer, so as not to overload them. And we need to allow our design to emerge, deferring design decisions to the last responsible moment, where we have learned the most and are best equipped to make the right decision. We also need to recognise when to break out in the light of new information, and reconsider the big picture, either as an individual or as a team exercise.

These skills may not fit neatly into the stratified organisational structures of our current software development organisations. But the Uber programmer doesn't fit into this world either. We need to be producing more Uber programmers who are creative, highly skilled, and massively productive. We also need to develop organisational structures where Uber Programmers can thrive. Cooperating with other programmers and other parties like the customer; all working together collaboratively as a single team with a single goal: working software. Common code ownership, and using code as a design comunication tool is aligned with this view.

We need to be creating more Kent Becks.


William said...

Sorry Paul.
My site has been having problems with the hosting, so my blog is weird right now. But I’ll post this in the blog as soon as I can.

Now, It seems there is a lot of writing while I was away, so I will try to be succinct.
In this post, I see you work again over the emerging design and you are still trying to understand why do we need abstraction. I agree with a couple of your lines, and the other ones, seems to me, we are just talking about different things called the same name.
I will try to define my concepts, so we are on the same page. And I will repeat parts on other posts, so all the other commenters know what I’m talking about J

Designing is not creating documentation or governance tools. Designing is the process of deciding how to solve the problem at hand. You decide big and small. Micro-Macro design. You decide where your expertise is enough and with the enough information at hand. Design is not guessing, it is not UML writing and it is not modeling. Furthermore, Design is not BUFD.

Architecture is not decisions. Architecture is the actual structure of the solution, its elements and relations, internal and external. So, architecture is something all systems have, no matter it is documented or not, known or not, bad or good.

Extremes are bad (I’m not talking about XP particularly, of course). Over design, over modeling, over abstracting are not good things. So you should not do that.

You ask how do we know decisions taken are good without running the software? And then you said TDD helps you verify the code upfront. So I ask: How do you know your test decisions are good? If you write your tests wrong, you will probable get wrong decisions. If you tell me we have refactoring to solve that, or that it is assumed test is done good, my case for design is the same: we have refactoring to solve bad design and also we assume the design is good. If you assume decisions are wrong until you run the code, then you do not trust your teamwork.

Now, based on the definition above, I would change “design can emerge” to “architecture can emerge”. Your early or late decisions will make the architecture emerge, always. The question is how good is architecture with late/early decisions. I guess it depends on the actual decisions.

No, again, taking all the first stories (that statistically tend to be the most important ones since they are the first to come into mind), checking general points, studying market trends and making a decision on how to communicate is not guessing. Architects should not work like your say they do, simply reading a book and copying the solution, crossing their fingers hoping it will work the same in reality. Before making a decision, you must have all required information. And a strategic decision is a guide for micro decisions, not an imposed truth.

Modeling should not be done by only one person, It should be done by the whole team, having the most experiences taking the lead and the other learning in the process.

Agree with your learning topic. Nobody is saying, though, I create something the kid cannot learn from. All the contrary: the kid works with me, he needs to understand all the decisions, why did I take them. He can come with a fresher view and I will be the one who learns.

Architecturing, the process, is not a micro cycle thing. Architecturing, or the process I do, is not just deciding something, I do lots of things during the day, from programming to investigating, to coaching to teaching to discuss strategy to sale to clients, to learn from clients, to define standards, to review code, to review design, to fight against managers, Oh, and to write to blogs, among several other things!
Finally, organizations need to change. But getting rid or architects is not the solution. We are allies, not enemies. At least me.
We need to create more Williams, then….


Paul said...

Hi William,

We do need more Williams :)

We are in total agreement. The only difference is the title. What you have described here is Software Development. So what is wrong with the title Software Developer?

You may not fit the picture I'm describing, but unless I've been in a dream the last 18 years, the picture I describe is real and exists.

In my experience, the best most experienced developers act exactly as you do. Now all developers are not the same. I try to split developers into 3 levels. Level 1 is a junior person just out of college. Level 2 intermediate who perhaps has completed at least one major project and level 3 is a senior person who has completed a few projects and has at around five years experience and is capable of taking a leadership role. From Steves Yegges post on Noobs it looks as though I could do with one or two more levels. My level 3 is what Steve describes as the teenage phase :)

Now in my teams I try to keep a ratio of level 3+ people to the others of around at least 1:3. The thing is though we are all Developers all with an equal voice and an equal say. We work together in a team (scrum) all pulling together. There is no job too big or too small for anyone.

We all own the code, we all own the architecture, we all take part in design decisions. We practice common code ownership, common architecture ownership, common design ownership. We work in pairs. Rotating all the time and learning from each other. Now most of the time the lessons flow from level 3's to level 1's. But occasionally the lesson is the other way round, and we all have the humility to accept this and no one is too proud to learn and listen to anyone one else.

Leaders in the team emerge naturally, because we all know who is best at certain things. If I have a really hard OO problem then I and the person I am pairing with will go ask William. If the decision we agree as significance to the whole architecture, then we will call the whole team into a huddle and present the solution on the white board and get everyones consent.

As a leader my role is not to decide for others, but get them to make the best decisions for themselves. My role is not to tell people what to do. I listen and suggest things they may have not have thought of, allowing them to go through the thought process themselves and come to the right decision. This way they aren't dependent on me deciding the next time and they learn to make decisions for themselves. On occasion I will even knowingly allow people to make mistakes and learn from them, as long as it doesn't jeopardise the project and the success of the team.

I have worked with people who are much better qualified then I. Two guys come to mind, one of them has a PhD in Refactoring (yes it does exists). Yet he sat down and programmed side by side with our most junior programmers, he also knocked up Phython scripts to automate certain parts of our build process, he also led our morning standup meeting, suggesting when we should rotate pairs and he also coached and did all the things you do. PhD aside he was still just a developer on the team.

I have been fortunate to work with some very good people (lets call them level 4 and 5 Developers). The thing they all had in common was a degree of humility that allowed them to learn and a willingness to roll their sleeves up and get stuck in with everyone else.

One team, one role, one purpose. Teams that work on this premise have a certain feel to them. From the description of what you do it looks like you agree.


Paul said...

Hi William,
I said we agree on everything. Well not quite everything...

I don't believe I am having a problem with abstraction. I just prefer to stick to concrete examples. I find it helps to keep your feet on the ground.

You ask how do we know decisions taken are good without running the software? And then you said TDD helps you verify the code upfront. So I ask: How do you know your test decisions are good? If you write your tests wrong, you will probable get wrong decisions. If you tell me we have refactoring to solve that, or that it is assumed test is done good, my case for design is the same: we have refactoring to solve bad design and also we assume the design is good. If you assume decisions are wrong until you run the code, then you do not trust your teamwork.

I would say that you know your design is of good quality when you get concrete feedback that it is so.

Tests are just one type of feedback. Code smells are another, a drop in velocity indicating that it is getting more difficult to implement new features is another. A growing bug list is another. Customers finding the software difficult to use is yet another...

My point is closing the loop and using feedback. Inspect and adapt. Using a biological analogy I am talking about evolution and the survival of the fittest. For this to occur you need to punctuate the process with inspection points (the life and death cycle). For TDD it is every time you run your tests, for velocity it is every iteration, for customer feedback it is every time you demo the software, or better still release it to production. The more inspection points the faster you can evolve.

So Emergent design is not a single one off event, it is continuous I believe we agree on this. But it is not plucked out of thin air either. Feedback will tell you what the design needs to be. Assumimg you sart with the simplest thing, skill is having the ears to listen and being intimate enough with the system, the users and the system context to recognise the signals.Only then can you effectively inspect and adapt

When Winston Royce wrote his paper on waterfall development, he said the same thing. Analysis, Design, Programming, Testing are all concurrent activities. He merely represented them as steps to show their relationship to each other. From this we managed to get the idea that they are separate distinct activities. I am saying that they are not:

I would also say, that modelling at all levels of abstraction should be done by the same person. Remember that code is merely a human readable representation of the computational model, and it is the computational model that is our final program.

Arun said...

Paul, hear, hear!

I am cocooning - seeking out and reading what I already believe in. You do articulate quite a few of my thoughts.

It is indeed sad that our profession has been denigrated by the masses of people claiming to do it. I may blog about this myself sometime.

Reading the dialogue between you and William makes me align more toward your viewpoint. However, there are some very good points that William makes that need to be answered.

First among those is the fact that creating a test now begins to take on the central role. I agree that it is a good thing, but we have to realize that we now need to apply our learnings in software development to the development of tests.

Second, is his call for the need for some amount of reflection and consideration before beginning on a journey, and also at points during the journey. One of the big reasons for this is that the cost of change has not reduced in all aspects of software development.

I believe that the single aspect that makes agile approaches possible is the reduction in the cost of change. However, this cost is still high in a number of very crucial and fundamental areas.

It is not possible to cheaply refactor some decisions around persistence. Likewise some decisions around messaging.

There is no reason that TDD cannot be supplemented with some amount of considered thought and reflection at various points in the development process. The two approaches can and should complement each other.

Paul said...


Sorry for leaving it so long to get back to you, but you pick up on a couple of points that I agree need to be addressed:

First among those is the fact that creating a test now begins to take on the central role.

I agree. Tests in themselves can lead to a myopic view of the problem. At the beginning what we want is a vision and a broad set of goals. Very quickly though these things need to become concrete and at this stage they need to be testable. This is nothing new, and is no different from "testable requirements". If we can't express our problem in an unambiguous testable way then how will we know when we are done?

So by test don't think at the level of JUnit. It can be a customer acceptance test written in plain English as part of a user story. Better still it could be a Fitness test (requirements by example) that can be automated.

Second, is his call for the need for some amount of reflection and consideration before beginning on a journey, and also at points during the journey.

I agree with this. Emergent design is not a mechanical process. I make the same point too. I am just making a different emphasis. You can "blue sky" all you like, but at some stage you need to make concrete decisions, based on concrete requirements and concrete data. In a lot of cases it is only the decision that is concrete, whilst the goals and the data upon which the decision is made is vague. Emergent design avoids this by compelling you to defer making decisions until you have concrete data to base them on. Your instincts are fine, but are not sufficient. They need to be questioned and tested.

An easy remedy to speculative decision making is to require that the any design decision is expressed as a test. Tests can be readily tested to see whether they align with the requirements (all requirements can be expressed as tests and visa-versa). If the tests don't align, then the design decision is unsupported by the requirements.

So ideally in Emergent Design, all design decisions should be tested and justified in a concrete way.