Pull is the new Push

Just finished watching the video of Kent Becks keynote speech at the recent Lean Startup Conference. First thing to say is that I've got a lot of respect for Kent Beck. The thing that grabbed me about this talk though was that he was advocating that we could use pull and flow to accelerate learning in the context of an internet startup. The learning is identifying where the value is. What is it that customers are willing to pay for, and in turn will mean that your startup will become financially successful.

I'm a forty something, and I remember when none of this technology existed. I remember when it all was a promise. The microchip was going to change our lives for the better, by giving us all lots of leisure time to spend as we like. Well it didn't quite work out that way did it? Instead technology has made our lives faster and less secure. So given this back drop who is going to willingly turn to technology as a fun thing to do?

Sure people will use technology to interact with other people. I guess that's exactly what I'm doing now, but few will be interested in technology for its own sake. There appears to be a dilemma here. People enjoy interacting with other people, yet increasingly they find that they need to put up with irksome technology in order to do so. The village pub were everyone knows your name is now a thing of the past. Instead we're all sat at home by ourselves trying to make friends through blogs and chat rooms.

Is this a trade that anyone really wants? I doubt it. I'd much prefer being in a real social setting right now, getting this all off my chest :) So the trick it seems is to find ways that are less irksome then others that allow us to interact with each other through these horrible technology mediums we've invented.

The village pub closed down years ago by the way, and even if it hadn't your pals haven't the energy to come out for a drink, because the computer at work means that they now have to do three peoples jobs, so in the evenings they barely have the energy to sit through a TV dinner before rolling into bed :(

Kent is pretty open and honest about his failures in startup land. I think he is being slightly unfair to himself though. No one wants this stuff. People will put up with it only if they really have to, or if they see a way of making money out of it. Technology in the work place is seldom a win win proposition. Often a minority gain whilst the majority loose. Is it any wonder that technology as a leisure pursuit is a hard sell?

The whole drive for being "Lean" and efficient leaves me a bit could. We seem to think that by speeding everything up we can some how make the world a better place. I like the idea that many things of value can't be measured and many of the things we can measure have little value. Try putting a number on the value of your humanity?

I can't help but think that all this technology is being pushed on us all. Kent is right, we can learn which aspects of the technology are less irksome to users, to the point that people will actually put up with it, in order to make contact with their fellow man. But is this really market pull? Or just a desperate means of pushing stuff onto people that deep down none of us really want?

An After thought:

Despite this rant I must admit that I'm still a bit of a techno-geek at heart :) I noticed myself salivating over a colleagues iPad the other day, so I'm still attached to the dark side :) What I'm talking abut here is really a sense of balance and purpose. People aren't machines, and their desires and needs have been built up over thousands of years of evolution long before technology existed. We all need balance in our lives in order to gain a sense of well-being, and our current technology driven society isn't providing that I don't think, which brings me to purpose. Technology is just a tool, its what we do with it that counts. Since the microchip revolution I don't believe we have given sufficient thought to the purpose we should apply all these cheap CPU cycles. Quicker and faster, more with less, isn't a forfilling answer, especially when it ends up turning our lives into one big frantic blur. I guess that's why the iPad is so attractive. It offers a glimpse of a technological future were the medium fades into the background and the content comes to the fore. Alan Kay's Dynabook is with us. Now what was Alan's purpose for the Dynabook again? Oh yes, learning and enlightenment. Lofty ideals, but perhaps we'll get there some day :)

Popularity and Success are not the same thing

After my last post, I went back and read through Giles blog post again. I missed his main point, and I dismissed what he had to say far too quickly. This video of DHH and Tim ferri, drove the point home for me and is well worth watching. This talk didn't prove as half as successful at Rails Conf, as Bob Martins.

Whilst I still think that Giles point is tangential to what Uncle Bob had to say, I think it is a profound one. That is, that any proclamation that requires free thought is likely to be unpopular. This lack of popularity doesn't equate to failure however, it just means that such ideas are more likely to benefit the few, rather then the many.

Uncles Bob's use of Smalltalk as the epitome of failure, rather cleverly raises the spector of fear, and as we all know fear is the great motivator of the masses. So if as a Ruby programmer, you want to be in a job for the foreseeable future, then make sure and do TDD. It's kind of like a Mom using threats to make sure that the kids eat their greens.

Whilst DHH and Tim and Paul Graham, are fundamentally right, they don't address the psychic of the average mainstream programmer, which is mostly dominated by the fear of not finding a job. So in a way this brings me back to were I started. Sometimes being right isn't enough. Sometimes you need to be effective too, and whilst fear is the worst of motivators, Uncle Bob is addressing the world as it appears to the vast populous, which is an effective tactic.

The world that is Giles reality is a rarefied place and the few people and languages that live there are likely to remain obscure for some time to come yet. The last interesting point is that it appears that DHH has out grown the community he created. This is hardly surprising. Like I say followers don't think for themselves, they rely on others, and as Rails becomes ever more popular, the community will become dominated by followers, just like what happened with Java.

For someone like David, this influx of non-thinkers must be nauseating.

Paul.

What Killed Smalltalk: Sometimes Balls can be Useful

Uncle Bob, has been his usual charismatic and entertaining self in his keynote speech this year at Rails Conf. He presented a dire warning for the Ruby Community: "What Killed Smalltalk Could Kill Ruby, too". So what is this beast of death? Well Uncle Bob blames the ease at which programmers could create a mess in Smalltalk as the thing that killed off the language, quoting no other then Ward Cunningham to back up his argument. Interestingly he makes only brief mention of another VM based OO language with garbage collection that appeared in 1995, and was market mercilessly and given away for free, whilst Smalltalk vendors were still charging £3K plus per pop for Smalltalk licenses (yes that's over £3000 per seat). Hmm...

Anyway as you would expect, the Smalltalk community is up in arms. James Robertson has this post. And Giles Bowkett makes this response entitled "What Killed Smalltalk: My Balls". Giles response, demonstrates IMHO complete ambivalence to what is actually happening in mainstream programming today. Whilst IMHO Uncle Bob clearly has his finger on the pulse. I have never been a full blown member of the Smalltalk community. I've always been one of those people on the side lines, using Smalltalk for fun and hoping to get a Job that would allow me to write Smalltalk commercially, but never being fortunate enough to do so. I think that gives me a more balanced perspective. I am neither a curly bracket or square bracket zealot, having lived most of my programming career with both. Whilst I've always admired the Smalltalk language, I have noticed that the Smalltalk community has a tendency to be insular and inward looking (with no sign of those balls Giles speaks of). Anyway, here is my comment in response to James Robertsons blog post. I thought I'd post it here (with some minor edits), since I know that very few people take the time to look at Smalltalk blogs:

There are two audiences here. One audience are the 20 somethings who were merely a lusty glint in their fathers eye when the clash of C++ and Smalltalk was taking place in the 1980's. The other audience are the people who where actually there.

For the 20 somethings, I think there is a lot in this presentation to get excited about. First of all it makes the point that good ideas can fade. It also makes the point that great languages provide the programmer with great power. With this power comes responsibility. As developers we have a choice. We can allow ourselves to be dumbed down and given dull tools so we won't hurt ourselves, or we can become responsible and disciplined to the extent that we can be trusted to use sharp knives. This message is an important one. The other great message, Smalltalk, which is now being discussed amongst a new young audience. If any of these young kids are half as curious as I was at their age, then we should expect downloads of Squeak to spike this weekend...

As for the other audience. The people that were actually there. It is easy to dismiss this presentation as a pile of BS. Well it mostly is, but to engage with it only on a factual level is to miss the point. This presentation is not for us. Bob Martin is not a Smalltalk programmer and never was. I was on one of his OOD courses in the 1990's and he asked who in here as used Smalltalk? I was the only person to sheepishly raised my hand. He then went one to castigate Smalltalk, saying that you couldn't use it for anything serious because it would lead to a mess once you got to more then 10K lines of code. Back in them days Bob was a curly bracket language zealot.

I'm glad that James has admitted that that back in the 90's the Smalltalk community suffered from arrogance too. I read Giles response and it sounded more like a tirade. The mainstream developer community at the moment is in flux. They trusted in vendors only to be let down. Now they are looking for new leaders. Most of them are followers, if they weren't they would have self educated themselves long ago, and Uncle Bob wouldn't be able to get away with his historical inaccuracies. The Smalltalk community are in a great position to provide leadership. The Agile movement came from Smalltalk, TDD from Smalltalk, Ruby from Smalltalk...

So why isn't the Smalltalk community doing so? Why wasn't a Smalltalk programmer giving this keynote instead? Where is Alan Kay when you need him or Dan Ingalls? Is it because we've all got a reputation of being grumpy old men? It is up to the Smalltalk community to make themselves relevant to these new young kids. Bob is doing his best to tell a story second hand. The Smalltalk community should be telling this story itself. Looking in the mirror at yourself and your own short comings, is much harder then pointing fingers at others.

Paul.

The road to mastery - Practice, Practice, then Practice some more...

The craftsmanship movement has hit the ground running. First a conference in London organised by Jason Gorman. Looking through Jason's Blog, and from the video, it looks like the whole Xtc London crowd were in attendance. Jason is an established member of the Xtc crowd, so I suppose it shouldn't have been surprising. Familiar faces like Ivan Moore and Nat Pryce. It brings me back to my early XP days, when I too attended such conferences and the excitement and delight of it all, not to mention the friendships and the insights. What fond memories.

So after going full circle, the "Agile" movement are now back were they started: Writing Code! Or should I call it the post Agile movement? Or better still lets just call them the folks who like to program, get stuff done, and take a pride in doing it well.

Micah Martin was there, and presented his Langston's Ant Ruby Kata. Now Micah's dad, Uncle bob is the guy responsible for teaching me the fundamentals of OO design. So anyone schooled by Uncle Bob, will probably have something to say that would prick my interest. Watch the video. Its a real expose of the Ruby language and the mechanics of programming in general.

Micah borrowed the word Kata from marshal arts. A Kata is an almost ritualised performance of a set of martial art skills, with the desired aim of re-enforcing those skills through practice. So Kata's are meant to be performed again and again, until the practitioner becomes use to the moves involved. Its a wonderful idea, and is synergistic with Ron Jefferies emphasis on practice and learning through doing. Ron's bowling game example could be considered yet another Kata. Hopefully Micah will develop this idea further and I'm looking forward to a whole catalogue of Kata's that the aspiring programmer can learn and practice. Katas to me seem like a great tool when going through the Shu (hold) phase of learning.

Focusing on Micahs Ruby Kata, what was interesting to me was his interpretation of BDD. I have been doing quite a lot of BDD lately, and I have found the style of BDD I've been learning a tad verbose. Micahs BDD style using Rspec is very similar to my own TDD style, and a lot more appealing as a consequence. Given that Uncle Bob learned TDD from Kent Beck, and I learned TDD from a bunch of XPers (Ivan Moore, Steve Freeman, etc) who presumably picked it up from Kent Beck too, its clear to me that Micah and I share a common BDD/TDD lineage. This raises an interesting point about style.

Micahs software shop is called 8th Light, and they conduct an apprenticeship program. Now I always thought of an apprenticeship as something for novices, but 8th Light take a different slant on the apprenticeship idea. Rather then an apprenticeship being just a means of teaching someone the basics, 8th Light also sees an apprenticeship as a means of teaching someone the 8th Light Style. This is another great idea. From my waterfall days, I have known that great programmers will get the job done, using what ever approach is familiar to them, and there is no one right way, No Best Practice, No one style.

Corey Haines explores the idea of style with Paul Pagel of 8th Light in this video. They also talk a bit about the recent Craftsmanship Manifesto and what motivated it. They suggest that programmers should learn more than one style during their Journeyman phase. Again this is something that is synergistic with my experience. I remember working with Ivan Moore and Erik Deorenenberg on the same project. Both great programmers, but each with a different style. Whilst I found Ivans "Smalltalk/Python" biased style more appealing, Eriks prolific nature and practical bias was extremely informative too, and I vowed to get my keyboard skills up to Eriks levels (which is still something on my to do list :)).

So the Journeyman must be prepared to break with his master and explore alternative styles. This seems to me like the Ha (break) phase of learning. At 8th Light they are hoping to facilitate this by organising exchange programs with other software shops.

So there we have it. People walking the talk, and software as a profession actually being practiced. It is interesting contrasting the Craftsmanship movement with the Kanban movement, both recent off shoots of Agile. One group (Kanban) are busy telling others how to manage software development, and are selling their services as Consultants, whilst the other group (Craftsman) are busy writing software themselves for real customers and sharing their practices and insights for free with anyone interested in learning.

I saw this on Jasons blog:

Q: How do you get to Carnegie Hall?

A. (craftsman): Practice, practice, practice.

A. (consultant): Pay a respected market research company shedloads of money to write a report that concludes you're already at Carnegie Hall, and then make a killing selling maps and directions (even though you've never actually been there)

I think it says it all. All's left for me to do now is to get practicing myself. I'll think I'll start with the Langston's Ant Kata in Newspeak. Now that should be fun!

Post Agile - Beyond Best Practice and towards Professionalism

The Agile community has been going through growing pains of late. InfoQ have been tracking this in a number of good articles referencing blogs posts by various members of the Agile Community.

I'm not going to reference all the blogs, but I think this introspection is a good thing, and I sense the birth of something new as a consequence. Bob Martin describes the ethics and the ethos that underlies this new thing in this presentation. He talks about a consensus arising around favoured software development practices, and he raises the notion of an accepted standard of Craftsmanship, which when tied with the ethical issue of actually delivering to our customers what they payed for, raises the possibility of software development actually becoming a profession. The idea of craftsmanship is further explored in this blog. I like the idea expressed here of "doing it right" rather then merely "getting it done". Andrew Walker talks to the ethical issue by making the point that we can't really in any honesty call ourselves a profession whilst the failure of software projects is still endemic costing our customers millions for no or little return.

Jim Shore has been busy blogging about the future of Agile also, the most infamous is this post which shines light on the problems that arise when people adopt Agile Management practices like Scrum, but choose to skip the more difficult challenge of adopting sound technical practices. The post by Jim I find the most revealing though is this one on Kanban. The reason why I find this post interesting is that it raises the question of what is "Best Practice".

Jim tries to explain why people are choosing to eschew iterations, calling planning meetings waste, and are choosing to shrink their backlog, calling it excess "inventory". You will recognise "kanban", "waste" and "inventory" as terms bored from Lean Manufacturing. Lean ideas are all the fashion at the moment it seems :). Besides the fact that these ideas are being applied out of context, labelling them "Kanban" (which interestingly is misappropriated since the term already has a specific meaning), reveals our desire to label and promote "Best Practice". In the comments Steve Freeman makes this point, and says that after speaking to David Anderson, one of the original people who invented what has become known as "Kanban", he found out that the reason why he did "Kanban" had nothing to do with eliminating waste, but rather it was a pragmatic response, to an incumbent waterfall culture that didn't allow for iterations. David himself later comments and confirms that Steves summary is accurate.

So one persons chosen practice in a given context, is marketed as best practice for a whole industry. Some in the Agile community have been clear about the need for self organisation and doing what works for you, but historically process improvement has always been viewed in the western world as the adoption and the enforcement of "best practices", and old habits die hard. As a consequence, selling "Best Practice" is now an established cultural phenomena in itself. The Agile Community as been as guilty of (mis-)selling bottled snake oil as all the other process improvement fads that have come and gone before it, despite the small print in the Agile Manifesto that speaks to the contrary. This opportunistic selling perhaps explains why the Agile banner lacks credibility in the eyes of many. They've seen it all before. Ethical issues aside, our culture of promoting "Best Practice" is very naive. Process improvement for an activity as complex, difficult and diverse as producing software, cannot readily be bottled and sold; which probably explains why we still aren't a true profession today.

A more recent article on InfoQ refers to a blog post that makes the point that Agile and Lean aren't the same thing. Now this is something that should be self-evident to anyone who as taken the time to read the work of Taiichi Ohno on the Toyota Production System for themselves, and shouldn't need explaining. Even so, I think that recognition that we cannot in all honesty promote Agile, Lean, Kanban or anything else as "Best Practice", devoid of context is a big step forward. The emerging consensus is that what really matters is developing competence in a useful toolbox of favoured practices, leading to a high standard of craftsmanship. This is not news, but it is nice to see this common sense point of view finally getting the top billing it deserves. It also perhaps signals that as an industry we are maturing, and now able to face the difficult questions that arise when you ask what does it take to be reliably successful at producing software. This fundamental question has hung in the air, ever since Fred Brooks first raised it. I'm tempted to go further and suggest that this change in the mood music may also be signaling the embryonic beginnings of professionalism. Or perhaps that would be going too far :)

What are your thoughts?

I don't know

If you read something new that is contrary to your current understanding you are faced with several possibilities:

1. The new idea stems from a new context which you haven't experienced. In this new context the old idea that you are familiar with no longer applies.
2. The new idea hasn't been rigorously tested, hence the need to seek corroborating testimonies.
3. The new idea has been corroborated, but the corroborators are involved in a conspiracy of lies in an attempt to deceive.

Now before you jump to option 3, you would rigorously explore options 1 and 2 first, for the simple reason that you may learn something new. Well at least you would think so.

In my experience this is seldom the case with programmers. In most instances when faced with a new idea that challenges current "best practice", I see programmers jumping to option 3.

This blog post and the comments that follow is a clear example.

So why is this? I think it has to do with how programmers process information. As left brained people, they build ever more elaborate models of understanding of what is "best". New information to the contrary challenges that model and rather then see the model fall like a deck of cards there is a strong impulse to reject the new idea.

I have blogged about this before:

Not knowing shouldn't be threatening, a state that should be extinguished quickly, with ever more elaborate models. No, not knowing is just part of the rich fabric of life. In fact there is power in the unknown. In acknowledging that there isn't a simple formula, order can be found in chaos.

Finding order in chaos is one of the most basic tenets of Agile Development. It is this philosophy that eschews any attempt to be deterministic and know in advance what is best. Instead Agile development embraces an empirical approach of trail and error, suck it and see, where not knowing and discovery is a natural part of the process.

This philosophy seems to be a hard sell. Everyone wants a fixed model, kind of like painting by numbers, which you can buy off the shelf and will solve your problem each and every time. In an attempt to fix the model scepticism is deemed as negative. The vice of non-believers who refuse to "get on message" about what is "best".

Keith Braithwaite takes a contrary point of view. Rather then seeing scepticism as always bad, he believes that healthy scepticism is a good thing, a sign that you are actually thinking and processing new ideas on their own merits. I agree. The problem with 'best practice" and a "fixed model" is that for any non-trivial process "best" is always conditional. So in different contexts what is best will differ. So what is right for one team may not be right for you. So there is no ready formula, and you need to discover what is best for each specific context, which means admitting that perhaps you don't know. This is why you can't master Agile development by just reading a book, and most Agile advocates suggest that you commandeer the help of an experienced coach who has experienced several different contexts to help guide you in discovering what is right for you.

Keith makes his point very well in an interview on InfoQ. At around 24 minutes and 49 seconds in to the interview, the interviewer asks "How do you choose the right Agile management tool if you are new to Agile?". To which Keith answers: "I don't know". From this admission you can actually see a process of discovery ensue. Keith then goes on to say that in lieu of knowing he would suggest using the simplest thing that works, an Excel spreadsheet. After using the spreadsheet for a while and writing macros to automate the calculations you needed you would learn what it is your process needed from an Agile project management tool and would then be able to decide which tool if any on the market best suited you.

It is a magical moment, with Keith shacking his finger in excitement as he discovers the answer to a question which he previously didn't have an answer to. The interesting thing for me is that the discovery started off with the statement "I don't know".

How many of us would respond to our boss when asked a question like "What tool should we use?" with "I don't know"? Very few in my experience. What is more likely is a long diatribe based on our prior context and what the book says or what we've read in reviews. As a consequence the opportunity for discovery is lost.

Take the time to watch Keith's presentation and try saying "I don't know" more often. You may find that it accelerates your learning tremendously.

Further Thoughts:

Alan Kay has another explanation for why people with prior context find it difficult to accept new ideas:

"Kay blames this lack of innovation on the fact that most adults employ instrumental reasoning to evaluate and apply new ideas. This means that adults have difficulty evaluating new ideas because they're carrying too many existing goals, and too much context to be able to see the full potential of new ideas. One way to overcome this problem is to work with children. Children won't know that something is difficult or that hundreds of papers have been published explaining the intricacies of some particular problem. They are more likely to experiment where adults would have given up, and to ask, "So what else can I do with this?"

I think this is similar to my "preserve the existing model" idea, where people reject new ideas that don't fit their established understanding. The same issue is also seen in science where breakthroughs happen during moments when some one is brave enough to go against the established orthodoxy. It is interesting that Alan Kay believes that children don't suffer from this. This fits with the proverb "you can't teach an old dog new tricks". So one solution is to try and maintain a child like curiosity and always ask the question Why? in the annoying way children do. I do this and it works well. If you ask why 5 times, it is usually enough to reveal something new.

Craftsmanship

Simon Brown has been re-evaluating the role of Software Architecture in response to the question:

Why do we need this new software architecture stuff? We've managed fine until now and projects *have* been successful.

Good question. Some teams don't dwell on the subject at all, any yet have produced remarkably well designed systems that have stood the test of time. Unix (Thompson and Richie), BSD Unix (Bill Joy) and Linux (Linus Torvald), just to name a few. In the case of Linux, there wasn't an opportunity to perform Big Architecture Up Front (BAUF) as a specific activity, since it was the result of the collaboration of literally thousands of developers on the web, so the Linux architecture we see today is truly an emergent entity devoid of a master plan.

What is architecture anyway? Simon concludes:

I say that architecture is about introducing structure and guidelines to a software project, and I've come to realise just how true this is. Without formalising these good ideas, a codebase with some otherwise great ideas can appear inconsistent, with the overall system lacking coherence and clarity.

In the discussion that follows, Simon goes on to explain that the architecture is "the big picture". Although he avoids using the word "design", I interpret what he says as architecture is the highest level system software design, which with a lot of programming languages is something that is difficult to gleam from just the code. I agree. What is disturbing however is that he flirts with the idea that defining this design is some how a different activity from programming, and calls for a dedicated role with dedicated skills. So architecture and consistency is something to be imposed onto a team of developers who can only code at the detail level and require an higher thinker, the architect, to impose "structure".

I like Simon, and I think he senses the irony in this. The team can't think and can only code, so the Architect must do their thinking for them. I wonder where this mindset stems from?

We will win and you will lose. You cannot do anything because your failure is an internal disease. Your companies are based on Taylor’s principles. Worse, your heads are Taylorized too. You firmly believe that sound management means executives on the one side and workers on the other, on the one side men who think and on the other side men who only work.”

Konusuke Matsushita – Panasonic

Simon can't be blamed. The values in our industry are such, that in many instances this is the only politically acceptable way to address the problem. People who are only paid to code and not think will struggle with architectural concerns and do need help. I pointed out that these same people will struggle with other aspects of programming too like listening to customers, effective collaboration and testing for example. Andrew Walker goes on to make the point quite eloquently that programming and architecting are one activity are can't easily be separated. I agree. In fact as I become better at my craft I no longer do BAUF (I once was an Architect in a past life) and I now allow my architecture to emerge as I write code.

So de-skilling the developer role by splitting out development activities like designing, communicating, listening etc into separate roles runs counter to the idea of good craftsmanship IMO. Craftsmanship is holistic. So how did we get here? Well de-skilling the workforce does allow you to pay them less and hire and fire people at will. It also puts those pesky all powerful "programmers" in their place, and may serve to provide the management with the illusion of control. There is a whole industry out there selling "Silver Bullets", and suggesting that software development can some how be automated and made simple by the latest Framework or piece of Middleware. And finally there are a bunch of ancillary industries like Business Analysts, Systems Analysts, Enterprise Architects, Solution Architects, etc, willing to do all "the thinking" at a fee, leaving the bulk of the team to only work.

Henry Ford had huge success with this approach in the 1930s, drastically reducing the cost of motor car manufacture by employing cheap unskilled immigrant labour. In recent decades however, in the face of competition from companies whose workers are empowered to think for themselves this Taylorist management approach has been found wanting. To see evidence of this you only need to open your newspaper. The Big 3 US Motor Car Manufacturers have recently gone cap in hand to congress for a bail-out. Toyota, Nissan etc build and sell cars in the US too, but they aren't asking for a bail-out! These Japanese manufactures are Agile enough to adapt to the needs of the market and are riding out the current economic storm. Toyota are even recruiting in the US for a new plant in California I believe. So why are these Japanese companies prospering at the expense of Ford, GM and Chrysler? In the 1950's Japanese motor manufacturers took a look at Ford, GM etc and rejected their Taylorist approach in favor of the ideas of Edward Deming. Deming felt that "scientific management" wasn't enough because it undervalued people and ignored human psychology. Could the Japanese choice of Deming over Taylor have something to do with the value they place on people?

Value and respect. I work as a Coach and I would never be as arrogant to suggest that I am the only person on the team qualified to determine the "big picture". I would like to believe that I have mastered my craft, but does that mean that I am the only person qualified to think? No this is arrogant nonsense and is exactly what Konusuke is speaking about.

Every human being can think and each of us should be encouraged to think. Our brain is our most valuable asset. The issue is that we aren't born knowing how to program well, and we all need to learn (even those of use who later become architects :)). So collaboration, architecting, coding, listening, designing are all things that must be learnt. The Japanese speak of Shu-Ha-Ri (Knowledge, understanding and skill), as the three phases of learning. A computer science degree will only provide you with some basic knowledge. To understand you will need to practice, and to gain skill will require repeated application in varying contexts. And of course you can't learn on your own. To learn effectively you need a teacher.

In past times when a man wanted to learn a craft he became an apprentice, and would live and work alongside a master craftsman for several years, receiving only food and lodgings for his efforts. After years as an apprentice he was deemed qualified and would become a journeyman, allowed to charge for his labours. A journeyman would then move from workshop to workshop gaining experience as he went. The best journeyman could become master craftsman themselves, setting up their own workshop and taking on apprentices of their own. Along the way people not suited to the craft would drop out rather than waste time, ensuring that standards remain high, and people found crafts that they were suited to. So in the past the west had its own version of Shu-Ha-Ri it seems. In fact this tradition still lives on in professions, like Lawyers, Doctors, Architects (the building kind), Fashion designers etc. So it looks as though the software Industry chose an unfortunate template in the Manufacturing industries to model itself on.

I can imagine that Henry Ford, must have seen traditional Coach Building Master Craftsman as an extravagant waste of money and a threat to his profits. Surely the job could be split into a number of simple unskilled roles with much of the work automated. This proved to be true (for a while) for motor car manufacture, but the same approach has failed us when applied to software. Why? because software development is not manufacture, it is not repetitive, instead it is creative. It is new product development, akin to fashion design, car design etc.

Successful software development calls for master craftsman like the names I mentioned earlier: Bill Joy, Linus Torvald etc. These masters have a responsibility to pass on their skills to apprentices whose job it is to think and learn by example at the knee of their master. The Agile community have gone back to basics and understand that developing people is the only way to pull the software industry out of habitual failure. People over process and tools.

Infoq has been tracking the tour of a Journeyman as he visits various workshops in an attempt to learn from the masters. His blog posts make interesting reading. Especially the way the learning occurs: side-by-side at the keyboard whilst programming. The master and the journeyman discussing the work and lessons being learned on both sides. An age old tradition, you can imagine Michelangelo working with his apprentices and journeyman in the same way.

Looking at the videos and listening to the interviews it seems such a natural way to learn, and the ideal solution for teams that are struggling with architecture or anything else. Hire the best people you can afford, and have them mentor their colleagues in a learning environment.

So obvious, yet Kevin a colleague of Simon seems to have missed this possibility. The idea that programmers can be helped to think for themselves escapes him as a possible alternative to the Architect going in and doing the thinking for them.

Perhaps Konusuke Matsushita is right. We do have an internal disease.

Object 101 - What is an Object?

Judging from the responses to my last post on uniformity it looks as though I started off with the bar too high. So lets lower it a bit. Lets go right back to basics and try and define what I mean by Object.

Concept

Firstly, the idea of Objects is a conceptual one. Alan Kay and his team did research for over a decade trying to understand how best to structure computer programs and settled on the idea of Objects. The concept of Objects can be explained by taking a biological analogy. Alan Kay speaks of the idea of an identifiable cell, which has a membrane encapsulating its insides from the outside world. Each cell is autonomous and goes about its job independently from other cells, but cells do collaborate in a loosely coupled way by sending (chemical?) messages to each other.

So Objects are analogous to cells and the key defining characteristics of an object are identity, encapsulation and messaging. Notice I haven't mentioned classes or inheritance. These things are merely just one approach to implementing objects and defining what goes on inside the membrane. There are object orientated languages like Self which eschew classes all together.

Lets explore these fundamental characteristics in more detail:

Encapsulation

The power of encapsulation is that it hides the implementation from the outside world. In the same way that a cell membrane hides the inside of a cell. The outside world need only know the message interface of an object (known in Smalltalk speak as the message protocol). So when I send a message to an object, I have no idea how that object will process that message. The object is free to process the message in anyway it likes. This leads to the idea of polymorphism, which is a Greek word meaning "many forms". Since an objects implementation is encapsulated behind its message interface, the implementation can take any form it likes. This means that message implementations are free to perform any side effect they wish as long as they satisfy the message interface. Notice again I have not mentioned classes or subclassing. Again subclassing is just one approach to achieving polymorphism. Any object that satisfies the message protocol is viewed as a polymorphic variant, and can substitute any other object that shares the same protocol.

Messaging

During their research Alan's team looked at a number of ways of getting their objects to communicate with each other in a loosely coupled fashion. If you take the biological analogy to its extreme, then each object should be totally autonomous and share nothing with the others. This means that each object should have its own cpu, its own memory, it own code, Alan Kay as even argued that each object could/should have its own IP address as a global identity. So an object in this view of the world is a networked computer. OK how to hook these objects together. Well the natural solution is asynchronous messaging, just like you get with e-mail. Since each object has its own cpu it doesn't want to block and wait until the receiving object processes the message. So an object can send a message without blocking and the receiver will send back an answer into the objects inbox its own good time. This approach is what we call the Actor model today, and as someone kindly pointed out, Alan Kay first explored this approach in an early Smalltalk back in the 1970's. Interestingly the Actor model has come back en vogue with Erlang which has adopted it to implement "share nothing" concurrency. This is why some people say that Erlang is Object orientated.

The share nothing approach to objects could be considered to be a bit inefficient. I'm not sure of the history, but Alan Kay and his team decided to move on from asynchronous messaging to a more restricted synchronous approach. With synchronous messaging all objects share a common processor (or thread) and message sends block until the receiving object has completed processing the message and has responded with an answer. This is the messaging approach that was settled on in Smalltalk-80 and released to the world in 1983.

State and Behaviour

So we now have objects sharing cpu, but each object still encapsulating its own memory (program state) and its own code (behaviour). The state and behaviour of an object is private (encapsulated). So what happens when two objects share common behaviour, but have different state? As an example what if I have two bouncing balls, one red and one blue? Do I implement two objects separately, duplicating the code? Obviously there is an opportunity here for these two objects to share common code.

As part of the private implementation of these two objects, sharing code is desirable ( the DRY principle). One approach is to create a new object to encapsulate the common behaviour. In Self they call this a trait object. This leaves the two original objects just containing state (red for one and blue for the other). Common messages sent to the red ball and the blue ball (like the message 'bounce') are delegated to the shared trait object (through something called a parent slot) which encapsulates the common code. The red ball however may decide that in addition to bouncing, it can blink too. Blink meaning changing colour from red to white and back again repeatedly. So in addition to 'bounce' the red ball adds the message 'blink' to its message interface. This behaviour is not shared with the blue ball, so the red ball will need to have its own blink code which is not shared. So in Self objects can choose to share some behaviour or may choose not to share any behaviour at all.

In Smalltalk-80, the idea of not sharing implementation was relaxed, although conceptually the idea is still useful. So in Smalltalk-80 all objects share behaviour with other objects of the same kind, leading to the idea of classes of objects. Again I'm not sure of the history, but I believe this is a more efficient approach then the approach adopted by Self (Self came about in the 1990s many years after Smalltalk when memory was cheaper and CPUs faster). So in Smalltalk all objects share common behaviour with objects of the same kind through a common Class object.

Classes

So we finally get to the idea of a Class. A class is merely an implementation convenience, and unlike what the C++ proponents would have you think, the idea of class is not central to OO. In the same way that objects can share behaviour through a common class object, so can class objects share behaviour through a common superclass, leading to the class hierarchies we are all familiar with and tend to associate with OO programming.

Notice I use the term Class object. In Smalltalk a class is a factory object. Incidentally a lot of the so called OO patterns in the Gang of four book are really C++ patterns. So for example there is no factory object pattern needed in Smalltalk where you get factory objects for free.

A factory object is an object that creates other objects. In Smalltalk such objects are stored in a global hashmap called Smalltalk and are available throughout your program. Global objects in Smalltalk are identified with a capitalised first letter in their name by convention. Classes in Smalltalk are just global factory objects and hence all have a capital first letter. This convention has been carried forward into C++ and Java.

So how do you create a class? Well you send a message to another class you wish to subclass. It will come as no surprise to know that the message is called 'subclass'. This message answers a new class. To this class you can add class methods and instance methods again by sending messages. Instance methods are methods that you want to appear on objects that are created by this class (remember a class is a factory), class methods are methods that belong to this class itself and defines the class's behaviour. Also you may want to add class and instance variables to your class to encapsulate state.

So how do you create an object? Well you send a message to the factory responsible for generating the kind of object you want. This means that you send the message 'new' to the class object.

This is where languages like Java and C++ take a cop out. 'new' in such languages is not a message send on an object, instead it is a keyword in the language. This means that you cannot override new and hence the need for the gang of four 'factory object pattern' in these languages.

Back to Smalltalk. In response to the message 'new 'the class will answer a new instance object. So now we have a new object. Incidentally we skipped over how objects are created in Self. In Self any object can act as a factory and is able to create a copy of itself. So in Self you send the message 'copy' to the object you want to copy. The copied object is now a prototypical instance which is why Self is called a prototype based OO language (rather than a class based OO language).

MetaClass

Back again to Smalltalk. We have factory objects (classes), and we have instance objects (objects), but where do the class methods live? Where for example is the code for 'new'? As I said before a class has two roles, one to define it own behaviour (such as defining new) and the other to define the behaviour of its instance objects. I called its own behaviour class methods. This behaviour belongs in another object, the classes class (classes have a class too). I think we need an example:

aTranscriptStream := TrancriptStream new.

The 'new' message implementation is defined in the class of the TranscriptStream Class object. The class of 'TranscriptStream' is called 'TranscriptStream class' which is also known as the meta-class. 'TranscriptStream class' also has a class: 'TranscriptStream class class', and so it continues. 'TranscriptStream class' and 'TranscriptStream class class' etc are all implemented as the same object, the Meta-class object (otherwise we could go on for ever). This circularity is one of the beauties of Smalltalk. Meta-classes do not exist in C++ and Java, and hence why 'new' is a keyword in these languages.

So the 'new' message is sent to the TranscriptStream object (which is a class) and the implementation is defined in the 'TranscriptStream class' object (which is a meta-class). Now how do we end up with 'Transcript'? Remember that in Smalltalk globals all start with a capital letter. To make something global I need to add it to a global hashmap called Smalltalk along with a global identifier (symbol):

Smalltalk at: #Transcript put: aTranscriptStream.

Then I can do this:

Transcript show: '3 is less than 4'.

Uniformity

The message 'show:' in the previous example is sent to the Transcript object (which is an instance) whose implementation is defined in the TranscriptStream object (which is a class). Interestingly you cannot tell whether Transcript is a global instance or a class. I initially mistook it for a class in my discussion with Stephan until I looked it up. The thing is with Smalltalk is that it doesn't matter. Instances, classes and meta-classes are all the same thing, they are all objects. Smalltalk is uniform and everything is an object, which is where I started.

Updated 4/12
Replaced Transcripter with TranscriptStream. Instances of both these classes satisfy the 'Transcript' protocol, but in Squeak Smalltalk the global 'Transcript' object is an instance of TranscriptStream. I found this out by printing 'Transcript class' in a workspace. Another approach is to print 'Smalltalk at: #Transcript'.

Objects 101 - Uniformity

Some interesting comments to my last post have prompted my to expand more on what I believe is good OO.

For me getting to know good OO started out with being skeptical about Bad OO and saying "I don't get it". This is why I think I understand Joe. Healthy scepticism is a good thing, especially when something blatantly just doesn't add up. I could go into a rant about why programmers find it difficult to say "I don't know" or "I just don't get this" and blindly admire the “kings new cloths” even when the king is naked, but I'll leave that for another day :)

As an example of good OO let me reproduce the Smalltalk code example from my response to a comment by Paul Homer to my last post:

3 < 4 ifTrue:[ Transcript show: ‘3 is less than 4’].

Ok lets try and express this in a more familiar OO language, Java:

if ( 3 < 4) System.out.println(“3 is less than four”);

What is bad about this? Like Paul Homer pointed out the instructions (if, <, ..) clearly take precedence over the data (3, 4, ..) in true procedural style. Also there is only one object here "System.out", which given that it is a global static object is hardly an object at all. System.out.println() is no different then ( #include <system/io>) printf(). So the whole statement is not OO, it is procedural and would look pretty much the same written in C.

But Java is supposedly an OO language, so surely I can express this as objects. Lets try:

(new Integer(3)).isLessThan(new Integer(4)).isTrue(new Block {
void execute() {
System.out.println("3 is less than four");
}});

Ok. I've created a DSL here in Java to get rid of the primitives and procedures and express everything uniformly as objects. So what is so bad about this? Well it doesn't read half as well as the Smalltalk example. All those parenthesis and periods tend to obfuscate the intent. Secondly I would need to write my own Integer and Boolean classes, overwriting the ones in the Java standard library. Java's Integer doesn't understand the message 'isLessThan" and Java's Boolean object doesn't understand the message "isTrue". Also the use of an anonymous inner class to simulate a block seems rather verbose, but without closures what else can I do?

So writing pure OO code in Java is difficult to say the least. Does this matter? Well if you are trying to learn OO with an hybrid procedural/OO language, then I think it does. I for one (using C++) definitely found it a challenge.

What do you think?

Paul.

Why Bad OO Sucks

Anyone who as read my blog knows that I am an OO advocate, but I was watching Joe Armstrong the other day of Erlang fame and he posed the question: "In which Object should you place a given function? To me the choice seems arbitrary". Now if you've got a solid grounding in OO design principles then the answer to this one is easy. Keep the function near the data. So the object with most of the data is where the function should be. I have posed this very same question at interviews and most of the time supposed experienced OO Java developers are clueless.

The fact that a language designer is asking this question in itself is interesting, and says a lot about how OO has been misrepresented over the years. Joes critism is thoughtful. He outlines why he believes OO sucks in a blog post. He then asks the question, why did OO become so popular? I repeat each of his reasons here:

Why is (Bad) OO so popular?

• Reason 1 - It was thought to be easy to learn.
• Reason 2 - It was thought to make code reuse easier.
• Reason 3 - It was hyped.
• Reason 4 - It created a new software industry

I agree with all of Joes points.

Reason 1- Bad OO is easy to learn because it requires no learning at all. C with Classes is still C. And a class can be used the same as a struct if you choose. The whole ORM industry is built on using classes as structs :) So a bunch of programmers have transitioned from C to C++ to Java etc without learning almost anything at all.

Reason 2- Code reuse as always been speculation. The languages where object reuse has been achieved are not the popular ones. Take a look at NextStep. They managed to produce a pretty reuseable Financial Framework in the 1990s, but it was all written in Objective-C. The "industry" had decided at the time that C++ should be the OO language of choice and went off and tried to build "Pink", "CommonPoint", "OpenDoc" etc which all failed. Why? Could it be that C++ is a static language and the idea of "live" resuable Objects only works with a dynamic language? Even with dynamic languages reuse is illusive. Why? Well back to the idea of categorisation. No two things in this world are truly identical. So my Banks idea of a Bank Account is different to your Banks. So even with the binding problem solved by using a late bound dynamic language, it doesn't mean I can send my Account Object to your Bank and expect everything to work. Proper OO aficionados know this and for this reason alone do not seek to reuse objects out of context.

Reason 3 - Hyped. No arguments here :)

Reason 4 - OO didn't create an industry. The IT industry created an industry. In fact the father of OO Alan Kay has spent the last twenty years bemoaning what the IT industry has done with his baby. Reasons 3 and 4 go together. The industry hyped OO with promises of reuse etc and then felt obliged to go off and build middleware to deliver on the hype. The astute amongst us noticed pretty early on that hey this stuff is getting pretty complex and heavyweight. The rest merrily jumped on the bandwagon with COM, CORBA, J2EE etc. So as Joe rightly points out the industry created a problem of its own making then went about selling us stuff to solve it.

At this point I should go on to defend good OO. But Joe wasn't speaking about good OO. Good OO doesn't suffer from all these problems and good OO isn't popular. Infact good OO is still in obscurity and suffering from funding problems, something that cannot be said for Bad OO.

No, I have plenty of other posts on this blog that speak to the qualities of good OO. And I am not going to say that good OO is intrinsically better or worst then good FP. I would just say that they are different approaches to modeling, each with its own sweet spot. And as always the thing that really matters is the fleshy blob behind the keyboard :)

Small is Beautiful

Computers are getting faster and more powerful all the time, yet whilst the average mobile phone as more processing power then all the computers NASA used to put man on the moon combined (well I'm not sure whether this is true, but if it is it wouldn't surprise me:)), our software of today is still no better then Doug Englebarts in the 1950's.

So why are us programmers having such an hard time keeping up with the hardware? The root of the problem is complexity. Whilst hardware has got much faster, it still performs a small number of very simple things like ' load', 'add', 'subtract', 'accumulate', 'shift', 'store', etc. It is the sequence in which these simple things are performed and the speed at which they are executed that gives the illusion of the computer being clever. In actual fact the computer isn't clever at all, the clever part is arranging these simple instructions into a useful sequence, and this is where programmers come in.

So programmers are the clever ones. Yes, but our cleverness is finite. Evolution works at a very slow pace and cannot be expected to keep up with Moores law :) So how can we expect to utilise ever more powerful computers if our brains just can't keep up? Well the starting point is realising that people are indeed the weakest link. Our cognitive abilities are finite and we need to organise our programming activities in such a way to best exploit our inherent strengths and respect our limits as human beings. This leads naturally to the subject of programming language design, but I'm not going to go there today.

No what I want to speak about is chunking and the magical number 7 plus or minus two. The way the theory goes is that us as human-beings can hold around 7 +/- 2 things in our brains at one time. Once the number of parts in a system become larger then this, then we are well advised to abstract and create hierarchies of things, ensuring that the 7 +/- 2 cognitive limit at any given level in our hierarchy is observed.

Anyway, back to software, 7 +/- 2 doesn't only apply to the organisation of computer programs, it also applies to teams. Small software teams tend to be able to adapt to changing requirements in away that large teams just can't. As a strong believer in Agile development and Emergent Design, this ability to turn on a sixpence is invaluable to me, and I tend to avoid large teams.

Experience has proven to me, that when it comes to software development that small is beautiful. After proclaiming this for many a year I have finally stumbled on the original paper that explains the theoretical underpinnings of this point of view:

The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information
by George A. Miller.

Enjoy.

Shu-Ha-Ri (Knowledge, Understanding and Skill)

InfoQ has picked up on a couple of blog post by Jim Shore recently. One where he speculates that Kanban and Lean approaches are viable alternatives to methodologies like Scrum and XP. Another where he says that Agile is in decline.

All the terms in italics in the above paragraph are just labels. Labels are just a convenient way to categories things. What we all would do well to remember is that in the words of Hayakawa: "the label is not the thing". Just because something is labeled Agile doesn't mean that it is an actual instance of the thing the original advocates had in mind. The Japanese understand this, and have developed a method of learning where they recognise different levels of mastery over the thing that is being taught. Shu-Ha-Ri, or roughly translated: Knowledge, Understanding and Skill. Just because some one knows the label it doesn't mean that they understand the thing. And just because they understand a specific instance of the thing, it doesn't mean that they have the skill to improvise and create a new instance of the same thing in a new context.

Categorising things and applying labels to them is a practice that is full of pitfalls. I will be posting more on this subject in the near future.

America is not Voting

I came across this on the BBC. Now why on earth in the richest country on the planet are people queuing for hours just to exercise their right to vote? I am living in the US right now and watching the election as an outside spectator is very interesting.

Traditionally around a third of Americans bother to vote. In fact one of the biggest grass roots issues over here has been voter registration. So why the apathy? This time around voter turn out is expected to reach record levels. So it looks like the silent majority have finally found their voice.

Central to the idea of democracy is the responsible citizen making an informed choice. In the UK we take our voting responsibility very seriously indeed. Which is as it should be given that people have died defending our right to vote. The whole ethos of our (free) education system and our public service cultural institutions like the BBC is geared to producing well rounded and well informed citizens able to utilise their vote intelligently. Over the years this has led to consensus politics, where there is broad cross party agreement on a number of major issues. When you have vice presidential candidates who can't tell you which newspapers they read, doesn't that say something about the strength of your democracy? If the politicians aren't informed, what are the chances of the electorate being informed? Isn't this the root cause of the polarisation that is so self evident in US politics today? The informed versus the uninformed rather than left versus right?

I picked up this quote from a documentary I once saw on the plight of the American Indian. An old Indian chief on a reservation was asked what he thought of the white man:

"The white man has many great things, but he cares not whether his people are wise".

Apt words, which still have relevance today. Living here now, I can say that Americans are wonderful people and worthy of the claim to be the greatest nation on earth. With any luck they will make the right choices and begin to address the weaknesses in their democracy and become even greater still.

The defining moment in the election so far for me was Colin Powells contribution. Colin Powells America is a country I would be proud to be a citizen of.

Comprehending Large Software Systems

William Martinez has started blogging again. He as an interesting blog outlining the roots of emergent design. It is worth a read. One factor that limits peoples abilities to apply emergent design effectively in a team context, is the high degree of design communication needed across the team. Since the design is always changing, maintaining a shared comprehension of the entire system design across the whole team can be a challenge.

The XP Solution
Emergent design is the design approach adopted by XP, and like other XP practices emergent design relies on the presence of other complimentary practices which are cleverly woven together to reinforcing each other, the whole being greater then the sum of the parts. The practices that come to mind are common code ownership where every developer on the team "owns" all the code and hence is responsible for comprehending all the design. Another is pair programming, where the quality of the code is policed by continuous peer review and design knowledge is communicated across the team by rotating pairs. TDD helps by documenting the design specification as executable tests, the design specification is then kept up to date by refactoring the tests along with the code. Another XP practice is small teams working in a "bull pen" or some other type of congenial space. The small size reduces the number of paths of communication, so if one developer makes a design change he only needs to broadcast that change across a small number of paths to ensure that a fully connected network of design knowledge is maintained. The congenial space facilitates 'osmotic' communication, where the communication of small day-to-day design changes and refactors occurs coincidentally, merely by people over hearing design discussions amongst pairs of programmers, or perhaps by the team breaking out and quickly discussing a design choice facing someone, coming to a joint decision, and going back to work with a new shared understanding.

Its All about People
With small teams seeded with people skilled in these XP practices, the XP approach to maintaining system comprehension amongst a team works very well. It does require a number of subtle skills though, many of which are right brained 'soft skills'. I usually recommend a ratio of 1 to 3, skilled practitioners to novices. With such a ratio I find that the novices pick up the idea pretty quickly. Interestingly Williams discussion on emergent design also identifies the need for teachers and learners, when it comes to mastering the skills needed, and of course we are all learners when it comes to discovering the emerging design of a new system.

Code as a Communication Medium
In my experience once you have experienced this informal means of maintaining system comprehension, you never really see the need for the types of traditional system documentation we have all gotten use to. Novices in these informal communication tools soon become masters themselves and go on to seed other teams. Unfortunately, in many environments we do not have the luxury of creating the right conditions for informal osmotic communication. So what then?

William made the following comment which is thought provoking:

Finally: Documents per se are not evil, but I have come to realize that the documentation tools may not serve the goals. I mean, I need to be able to take a 10000 feet look to a system, to see the big picture. But, I will not be able to see it looking at just code. A word document is of no help. I need something else.

The code is not sufficient? Why? I guess it depends on the prior context of the reader, and also on the quality of the code. In many organisations, people not directly responsible for the code are called upon to make comment and review 'other peoples' code. Now in this scenario the reader has very little context. The XPer would say, well you've broken the tenant of common code ownership. If the reader was part of the team, then he would also be part of the osmotic process and would have a great deal of context to draw upon when reading any given section of code. Yes, but lets assume that the environment is not conducive to "common code ownership". What then?

Readable Code
The other issue is the readability of the code itself. This can be a self fulfilling prophecy: people looking elsewhere other then the code to gain system comprehension, so the code itself need not be comprehensible. For XPers, their code is the main means of design communication, so they are very fussy about their code. For XPers production code and test code together encapsulate a body of knowledge that describes the design choices that they have made along the way. Kent Beck chooses to make a distinction between 'quality' code and 'healthy' code. 'Quality' code is code which has desirable external attributes, like a low number of bugs. Healthy code is code with desirable internal attributes, that allow the code to be maintained, changed and extended even when the team looses people and new people come on board. Healthy code is easy to comprehend and well designed. I don't make this distinction myself. I see health as part of quality, but it is interesting that Kent does make a distinction, perhaps in an attempt to raise the profile of internal code quality which in some environments is readily overlooked.

Domain driven design and the idea of an ubiquitous domain language which is carried through into the code, is yet another method to improve both the comprehension and the design of code. Cryptic names in code have long been seen as a barrier to comprehension, and lots have been written about 'self documenting' code and how to write code which is more comprehensible. So we do know how to write readable code. The real issue is whether readable code is truly valued by the team and seen as a necessity.

My experience is that after being given an overview of a system, perhaps at a white board, if the code is healthy and good package, class, method etc names are chosen and if the code as a good accompanying test suite, then with the help of an IDE, comprehending a system just from the code is possible. For me it normally takes an after noon or two and in the end I usually end up with a list of questions to be answered by someone who "owns" the code. Questions answered, I normally have a reasonable comprehension, and more importantly an understanding of the 'true' design , not what was originally envisioned by the architect at the beginning of the project, but has long since been superseded as the true design emerges during 'implementation'.

Other Documents
Having said this, two days of my time, and perhaps a day of someone else's is a lot to spend on comprehending a system, especially if I do not intend to work on it. If the code is not very readable it could take me a lot longer. It is ironic that the kind of organisation that does not value readable code is also likely to expect people external to the team to comprehend and evaluate the system in an instant purely from documents, forgetting that the code is the system and hence the most important document of them all. In many organisations this scenario exists so what then? I have found the use of a Software Maintenance Handbook very useful in the past. A guide for the explorer who needs to comprehend the system. A user guide, can be most informative too, especially when it comes to answering the question "what problem is this system trying to solve ?" But even with these documents, assuming that they are kept up to date, there will still be gaps in understanding, so what should you do?

Magical Tools
Lets assume that I had a magic wand that would produce the 10,000 foot view of a system at an instant, just by tapping my wand on the hard disk containing the code :) What then? would I really comprehend the system without speaking to anyone else? In my experience the answer is no. I would understand the structure of the system, the 'what', but I wouldn't know the 'why'. To know the why, I need to know abut the problem domain. In a complex domain space, my lack of understanding of the domain, would be a barrier to system comprehension in itself. So even with a magic wand there is no easy answers, and in the end you need to speak to some one, perhaps face-to-face.

Better languages
If your organisation doesn't allow people to take the time to explain the system to you, then perhaps the left side of our brain may be able to help out a bit. Domain driven design and object orientation both promise to help system comprehension through good naming and the use of modularity. Whilst some OO languages are turtles (objects) all the way down, most are not modular when it comes to high level 'architectural' abstractions. Newspeak is a new OO language that has borrowed ideas from Beta to allow you to define abstractions larger then a class. Once you have a language like Beta or Newspeak that allows you to capture high level design decisions in code, then you can use a code browser to project views of just the high level abstractions, filtering out all the details and providing the 10,000 foot view at an instant. Martin Fowler has been talking about such browsers under the label 'language work benches'. Martins focus has been DSLs and their comprehension and use, separate from the host language in which they reside, but the same idea can be applied to architectural layers, modules, components, libraries, packages etc, any abstraction that can be captured by your language and projected independently as a view. The Beta language already shows how this can be done, by allowing a graphical view of system components gleamed from the code. This idea should be familiar to anyone who has used TogetherJ, the Java/UML documentation tool.

Fix the root cause
As you can tell, my view is that system comprehension is often a people problem, born out of the ways people choose to organise themselves and the values and principles they choose to adhere to. The whys and wherefores for any given system should be tacit knowledge within the team that created that system. Tacit knowledge is born out of daily work and socialised informally across the team. It is only when we restrict the flow of tacit knowledge by creating artificial barriers within the team, or geographically splitting the team, or creating teams that are too large, that we create the need for formal communication. Formal intermediate work products are then handed-off from one part of the team to the next. These hand-offs are points of weakness and are best avoided.

Most software organisations are dysfunctional in this regard, by this I mean that their organisation is not well suited to their intended purpose, namely the production of high quality software. The solution? Change the organisation. I accept that most of us aren't empowered to make this type of change, but lets not forget that the root cause of poor comprehension is often self imposed.

Conclusion
So in summary, system comprehension is about communication. Ultimately the code is the design and hence the best medium for communication. With advances in language design the code can be made a better medium for expressing system structure at the 10,000 feet view, but comprehension is not all about structure, it is about meaning and purpose too, and to comprehend these requires a grasp of the domain. Gaining this knowledge often means talking to people.

As human beings competent in applying both sides of our brains, talking and listening as a way of comprehending a system should come naturally. When it doesn't then perhaps its a sign of a deeper problem, such as organisational dysfunction.

9th October 2008 - Expanded the section on 'Fixing the root cause' to clarify what I feel is the main barrier to system comprehension, poor organisation; in response to comments made by William Martinez.

Google Chrome - Vilan or the next logical step?

Its a strange feeling when an idea you think is gaining momentum suddenly becomes concrete before your very eyes, confirming that you were spot on.

It is not news that Google is a clear advocate of software-as-a-service, but I'm not sure that most pundits thought that Google would be making the bold step of creating their own client platform just yet.

Well Google Chrome is out and I'm sure that there is going to be plenty said about it. Is this the browser wars all over again? Well when I first read the news (on the BBC news website), I thought so, but when I took a closer look at the details released by Google I changed my mind.

As my last post on Flex makes clear, the current browsers as a client platform just aren't up to the job. The JavaScript interpreters are slow and the whole thing is single threaded, so poorly written JavaScript can lock up your whole browser. Flex has addressed some of these problems, but anyone who remembers Windows 3.1, would agree that Flex will never be able to avoid the browser equivalent of a Windows 3.1 blue screen :)

In the noise I'm sure that Googles words are likely to get lost so here is a link where you can read why Google chose to create Chrome and you can decide for yourself whether they are satisfying a real need.

Should we fear Google in the same way we use to fear Microsoft? I don't think so. This snippet from the Google cartoon on Chrome sums up their position for me (click on the image to make it full screen):

So they want to create something better, and we shouldn't be worried about vendor lock-in because "this better thing" will be open sourced. Over the years I have come to the firm opinion that innovation should never be held back by premature standards. Our current crop of browsers where originally designed to solve a different problem, and whilst they are familiar and use a standard metaphor, they cannot be said to address the needs of software as a service.

There is no reason why Googles new platform should not be open, supporting ActionScript and Flex along with ECMAScript (JavaScript). In fact given that Chrome will be open sourced, there is no reason why people can't write a Ruby or Python or even Smalltalk plug-in for Chrome themselves. This is all consistent with the RESTful idea of code-on-demand, which doesn't dictate what is meant by "code". It will be interesting to see just how open Google are with their new platform, and whether they provide extension points so that others can extend the platform to meet their own specific needs.

Infact if Chrome gains momentum, then other browsers like FireFox or Safari could choose to integrate Chrome, or even borrow ideas. We need to see what type of license Google chooses to use, but a BSD style license would allow both open sourced and closed sourced projects to embed Chrome code. Even if the code isn't directly re-used by other browsers then I'm sure the idea of a multi-thread/process, high performance, networked, client software platform will be.

Chrome seems like the next logic step to me, and the fact that it is coming from Google and is open sourced, compared to a closed proprietary solution like Apples Dashboard Widgets, is a positive step forward. We need to be vigilant and hold Google to account, but I think that Chrome should be cautiously welcomed.


Last point, in executing our responsibility to ensure that Google adheres to ethical competitive practices, why not clean up and clarify our language? I am not sure that the vague term "Cloud Computing" is helping, why not use the term "software-as-a-service" (as Google has chosen to do)? Or better still why not stick with the term "code-on-demand" as coined in Roy Fieldings REST dissertation? I believe language is important and labeling these new client platforms in a precise way will provide less wiggle room for marketers looking to exploit vague terms like "cloud computing" to their own ends. Code on demand makes the purpose clear and by using RESTful language, I think it will help to tie these new platforms into open standards like HTTP, ECMAScript and APP (Atom Publishing Protocol). Just a thought :)