Saturday, February 14, 2009

Post Agile - Beyond Best Practice and towards Professionalism

The Agile community has been going through growing pains of late. InfoQ have been tracking this in a number of good articles referencing blogs posts by various members of the Agile Community.

I'm not going to reference all the blogs, but I think this introspection is a good thing, and I sense the birth of something new as a consequence. Bob Martin describes the ethics and the ethos that underlies this new thing in this presentation. He talks about a consensus arising around favoured software development practices, and he raises the notion of an accepted standard of Craftsmanship, which when tied with the ethical issue of actually delivering to our customers what they payed for, raises the possibility of software development actually becoming a profession. The idea of craftsmanship is further explored in this blog. I like the idea expressed here of "doing it right" rather then merely "getting it done". Andrew Walker talks to the ethical issue by making the point that we can't really in any honesty call ourselves a profession whilst the failure of software projects is still endemic costing our customers millions for no or little return.

Jim Shore has been busy blogging about the future of Agile also, the most infamous is this post which shines light on the problems that arise when people adopt Agile Management practices like Scrum, but choose to skip the more difficult challenge of adopting sound technical practices. The post by Jim I find the most revealing though is this one on Kanban. The reason why I find this post interesting is that it raises the question of what is "Best Practice".

Jim tries to explain why people are choosing to eschew iterations, calling planning meetings waste, and are choosing to shrink their backlog, calling it excess "inventory". You will recognise "kanban", "waste" and "inventory" as terms bored from Lean Manufacturing. Lean ideas are all the fashion at the moment it seems :). Besides the fact that these ideas are being applied out of context, labelling them "Kanban" (which interestingly is misappropriated since the term already has a specific meaning), reveals our desire to label and promote "Best Practice". In the comments Steve Freeman makes this point, and says that after speaking to David Anderson, one of the original people who invented what has become known as "Kanban", he found out that the reason why he did "Kanban" had nothing to do with eliminating waste, but rather it was a pragmatic response, to an incumbent waterfall culture that didn't allow for iterations. David himself later comments and confirms that Steves summary is accurate.

So one persons chosen practice in a given context, is marketed as best practice for a whole industry. Some in the Agile community have been clear about the need for self organisation and doing what works for you, but historically process improvement has always been viewed in the western world as the adoption and the enforcement of "best practices", and old habits die hard. As a consequence, selling "Best Practice" is now an established cultural phenomena in itself. The Agile Community as been as guilty of (mis-)selling bottled snake oil as all the other process improvement fads that have come and gone before it, despite the small print in the Agile Manifesto that speaks to the contrary. This opportunistic selling perhaps explains why the Agile banner lacks credibility in the eyes of many. They've seen it all before. Ethical issues aside, our culture of promoting "Best Practice" is very naive. Process improvement for an activity as complex, difficult and diverse as producing software, cannot readily be bottled and sold; which probably explains why we still aren't a true profession today.

A more recent article on InfoQ refers to a blog post that makes the point that Agile and Lean aren't the same thing. Now this is something that should be self-evident to anyone who as taken the time to read the work of Taiichi Ohno on the Toyota Production System for themselves, and shouldn't need explaining. Even so, I think that recognition that we cannot in all honesty promote Agile, Lean, Kanban or anything else as "Best Practice", devoid of context is a big step forward. The emerging consensus is that what really matters is developing competence in a useful toolbox of favoured practices, leading to a high standard of craftsmanship. This is not news, but it is nice to see this common sense point of view finally getting the top billing it deserves. It also perhaps signals that as an industry we are maturing, and now able to face the difficult questions that arise when you ask what does it take to be reliably successful at producing software. This fundamental question has hung in the air, ever since Fred Brooks first raised it. I'm tempted to go further and suggest that this change in the mood music may also be signaling the embryonic beginnings of professionalism. Or perhaps that would be going too far :)

What are your thoughts?


victor xray said...

Have you seen Scott Bain's book "Emergent Design: The Evolutionary Nature of Professional Software Development"? It discusses in part what is a profession and what might be the features and qualities of a software professional practice.

On Amazon

sorry about the multiple fail above. ;-(

I have a short overview/review of it here on my blog (not my google blogger account) :

Paul said...

Hi Scott,

No, I haven't read this book, but I am very familiar with emergent design. Emergent design is part of the consensus of favoured practices amongst Agile practitioners. Others include domain driven design, TDD, BDD, common code ownership, pair programming etc. Basically the full XP toolset.

Now I don't think anyone is saying that you need to use all these tools all the time, but I think there is a growing consensus in the Agile community that you should be competent at all these practices and have them available in your war chest for when you need them.

I read your blog, and I agree with the indicators of professionalism you highlight. I much prefer the idea of craftsmanship though and serving and apprenticeship learning at the knee of a master craftsman, before becoming a journeyman in your own right.

The reason why, is that I believe that there is too much scope for misunderstanding and that the best way to learn is by example from someone who as already mastered the practices themselves.

This hands-on mentoring works to everyones advantage, because if you just aren't getting it, it will be clear to both you and your master, raising the question that you may be better off shifting to another profession.

As things stands you can sit in a corner, not speak to anyone, clock up several years experience and tout your self as a seasoned professional.

Getting away from a dry list of must haves is a better approach I think. I'd much prefer a written recommendation from someone who themselves has a solid reputation. Take a look at Uncle Bobs animated presentation (my first link) and you will see what I mean. Also Corey's blog talks about Craftsmanship too and serving an apprenticeship to journeyman and becoming a master craftsman in your own right.

At the moment this career path is all a bit speculative, but it works for Barristers and Surgeons, so why not programmers?

victor xray said...

Hi Paul

I get the feeling the 'profession' is still at the point of surgery in the early-mid 19th century - some people have discovered anesthesia, others still killing their patients with it; some using proper sterilisation, others completely disbelieving that washing your hands might even help.

Bain's book is well worth the read just for the chapters about professionalisation alone.

Learning by example, I've got no hesitations about, except to ask: beyond the most obvious people, how do we identify who the masters are? I think that's where we need to discover what the minimum basic set of 'professional' or craft disciplines actually are. I think we have good hints as to what the picture looks like, but not nearly the whole thing just yet - let alone full agreement across the industry.

How do we achieve that? I think it's a hard question.

Paul said...

>> How do we achieve that? I think it's a hard question.

Good point. I don't know. Thanks for the recommendation, I will buy Bains book.

My guess is the same way it happened for 19th Century Surgeons. Initially a group upset with the lack of professionalism, develop personal relationships and are able to recommend each other to clients as capable and competent. This is happening in the Agile community already. I work as a Coach, and I get all my business by word of mouth. If clients need good Agile people, I have a network of people that I can vouch for, knowing that each time I recommend someone my own reputation is on the line.

Overtime this group of like-minds, come together and form a guild. This is happening now on websites like

The guild, becomes a closed shop, where you can only join through invitation, and where you must pay tides. Eventually the guilds become full blown regulating authorities like the Bar (English Barristers) or The British Medical Society (Surgeons).

My point is that guilds came first based on personal relationships, and more formalised lists of entry criteria came much later (like hundreds of years later).

All historical of course. It will be interesting to see what actually happens with software.

Corey Haines said...

Great post. As you say, it is very important to stop thinking that one set of practices work in all situations. The direction towards school-based techniques that the Software Craftsmanship moving is heading encourages me that we might get out of the rut we are currently in.

The practices that I use aren't the important thing; instead, it is incredibly important that we USE practices and stop thinking that it is important for everyone to be 'pragmatic' and 'figure it out for themselves.' A lot of us were not and are not equipped for that yet based on our experience level.

Siddhi said...

I agree. Practices are just practices - you use them when you need to. Practices without context are meaningless. I don't think agile is about whether you use a practice or not, as long as it's there when you need it.

I've been noticing a shift in thinking in the community recently. Less and less people are talking about inspect and adapt, and more and more are talking about whether you are doing a suite of N practices or not.

No doubt it is triggered by a number of teams who simply drop practices because it is too inconvenient to implement. Even then, I'm not for the codification of a set of 'best practices' as a methodology.

I like the way Alistair Cockburn puts it - agile as a set of principles, which get executed as a suitable set of practices.

Merle said...

Loved this comment best ...

The emerging consensus is that what really matters is developing competence in a useful toolbox of favoured practices, leading to a high standard of craftsmanship.

I truly believe that teams will pick from the menu of agile tool sets and apply what works best for their context.

The next few years is going to get interesting as this all evolves.

A really well written, thought provoking blog.


Doyel mirza said...
This comment has been removed by the author.