Making programming pay

Building trust and unlocking the IT/ Business relationship - Show me the Working Software!

2011-07-02T14:31:00.000-07:00

Hoping to keep this one short. I've been reading a lot of Gerald Weinberg recently, and I must say that he as an exceptional way of cutting to the core. Computer programming as always been opaque. The people who pay for it, must trust to luck and throw their money into what must seem like a deep dark pit and hope that eventually they may get something back in return.

Invariably they get back very little. If it wasn't for the fact that this very little can go a long way, especially when it allows you to lay people off, business people would have given up on IT long ago. In a way, the wise ones already have. Astute business people don't rely on grandiose IT plans. They've become wise to the promises and have learn't to save their money by keeping IT simple.

Agile was meant to offer a new type of deal between developers and business folk. Give me a bit of money and I'll show you what I can do with it. And by "show", I don't mean a power point presentation, a lengthy requirement spec, pretty UML diagrams, or a fancy customer value model. No, what I'll show you is real working software. Something that you could potentially start using today.

Now this was a seismic shift in the tectonic plates. From a business perspective, software development in progress was no longer opaque. Well at least that was the promise. You can see what you are getting for your money whilst it is still being worked on. If you don't like what you see, you can choose to stop spending. If you do like, then you can choose to spend some more. You no longer have to sign blank cheques and hope!

Gerry Weinberg cottoned on to this psychological hurdle a while back (as early as the 1970's). Business people hate software developers. If they could, they would get rid of the whole lot of us (replacing us with cheap labour from India). And can you blame them? How many other industries do you get to sell faulty products late, and charge your customer for "upgrades" that merely fix faults you introduced in the first place? Not many. Making money out of our own faults, no wonder they hate us.

The IT industry have even managed to use this hatred to their own advantage. IT companies like IBM have learn't that you can sell business people the dream of getting rid or at least marginalising those pesky developers. Every IT silver bullet under the sun has been sold, and none of them have worked, yet they are still selling like hot cakes, SOA being the latest. A real example of the triumph of hope over adversity :) IT creates a problem and then sells phantom remedies. Now that's a real clever way to make friends and influence people!

Back to Agile. Wasn't the new deal meant to bring an end to all of this? Yes, but guess what? IT reneged on their side of the deal. It turns out that producing working software one incremental step at a time is really hard to do. Harder in fact then trying to deliver it all in one big lump, and IT weren't ever any good at doing that either. Now it's alright for the likes of Kent Beck and Ward Cunningham, to cut a new deal with business, because they have the wherewith-all to actually deliver, but what about the rest of us?

Chastened with accusations of Flaccid Scrum (run and hide in shame, being accused of being flaccid in any department is not a good thing :)), what have the mainstream gone and done? Well they haven't chosen to try and put some lead in their pencil. Oh no, instead they have offered every project management rouge they can think of (Scrumban, Kanban, Lean, ToC, ...) to try and mask the fact that they can't get it up (whoops, taking the analogy too far, I mean masking the fact that they can't deliver, but I guess for the ladies the two mean the same thing :)).

Now this is bad enough, but what is even worst is when these passing fads actually require the business to invest even more. This is what got me blogging. The idea that business people should spend hours of their precious time working with IT, analysing business value and prioritising work, when they haven't seen any frigging software worth it's salt in months. WTF?

Not content with mugging me, you now want to monopolise my time too, just to mask the fact that you haven't actually produced anything useful? Now that is really going to make me stop hating you - Not!

When are we going to wake up to the idea, that a deal is a deal, and actually start delivering on our side of the bargain?

Kent and Ward are right. Business people aren't so bad once you give them a reason to trust you. They will actually get off your back and stop micromanaging. They will freely engage and be generous with their time. They will pay for those memory upgrades you asked for 5 years ago. They will even think long and hard about the best ways to realise business value, but this is only after you've proven that you can reliably deliver working software. SHOW ME THE FRIGGING SOFTWARE!

Software people need to get over themselves and accept that they haven't got a good track record. The "new deal" presented yet another opportunity for us to get our house in order. To fix-up. This of course means acknowledging that we are in a mess in the first place, and I see far too few software people who have the humility to admit that!

Scala Bowling - A better Java or a Research Project?

2010-10-30T01:07:00.000-07:00

A short post in the hope that I'll get some input about Scala. I first looked at Scala when it first came about, around 3 years ago now I think. My first impressions were clever, conceptually robust, but schizophrenic. It is as though the language designers decided to shoe horn in every language feature they could think of. It was clear that it was way more powerful then Java, but for me it was too clever, too complex.

Well three years later I'm looking at Scala again. I've recently come back to Java, after spending awhile in C# land, and I'm sorely missing lambdas. A quick perusal of the usual Java forums convinced me of what I had already suspected. Java 7 and closures aren't going to see the light of day anytime soon. What Java needs is a re-write, and you could argue that Scala is it! So I downloaded a Scala plug-in for Eclipse, got "hello world" up and running and started pondering what to do next?

As chance would have it, trusty Ron Jefferies has come to the rescue again with his bowling game. His first attempt at Bowling with Scala is definitely imperative. A sort of Scala for Java programers style. As he anticipated the functional community were quick to come back at him with more functional suggestions, using pattern matching and recursion. Well Scala has all the functional paraphernalia so why not use it? There is a sort of unsaid assumption by the functional crowd that a pure functional solution is somehow intrinsically superior to an imperative one. As I mentioned in my post on Haskell the jury is still out for me. Sure state is evil, and where a functional style is a natural fit to the problem domain, then a pure functional style is a no brainer, but how often is that? My gut feel is that most of the stuff I tend to do (shuffling data between a database and a browser) is imperative in nature, well at least I tend to think of it that way. To me, pure functions are things you use in maths and best suited to mathematical analysis. I get the feeling that Ron also shares my misgivings, and yet again, the expert pure functional solution was found wanting when someone discovered a failing test.

So how should we use Scala? As a pure OO language where we exploit the benefits of immutability as much as we can? Or as a pure functional language were we exclude state entirely Haskell style? It was interesting to see that the expert pure functional solution came about as a direct port from Haskell. looking at the Haskell version and the Scala version, they are almost identical. Based on this evidence you could describe Scala as a pure functional language, that also so happens to support imperative OO programming.

The guys over at Object mentor may beg to differ. Their bowling examples, whilst functional in nature also exhibit an OO quality, yet they still make use of pattern matching. So their Scala style could be considered an OO/Functional hybrid. What is the budding Scala programmer from an imperative background to do? It seems to me that Scala is still a bit of an experiment, with people still working out the best idioms.

The option of using Scala as a better Java is always there of course, and if I get the chance to do so I will, but there will always be this nagging doubt of whether I'm doing it right? And whether a real Scala programmer would approve? I think languages should promote a style. C was very good at this, with everyone adopting the style of Kernnighan and Ritchie. Smalltalk also makes a very strong style statement, exemplified by the code in the Smalltalk-80 image. Scala doesn't seem to have this. It is though it is at least two languages, or perhaps three, all coexisting. A haphazard mix of paradigms, idioms and styles.

I guess I need to read the book by the inventor Odresky to see if he as a Scala style I can comfortably adopt. More important then the discomfort of "am I doing it right?", is the obfuscation of the problem domain that comes IMHO when applying a pure functional style to problems that are more naturally expressed imperatively. Should I always be living in fear of that next failing test that blows a whole in my cleverly contrived pure functional solution? Or should I be able to reason about the correctness of my program with a modicum of confidence by simply reading it?

I'm interested in finding out the thoughts of people who are further down the Scala path. All comments and pointers welcomed.

Pull is the new Push

2010-05-03T07:51:00.000-07:00

Just finished watching the video of Kent Becks keynote speech at the recent Lean Startup Conference. First thing to say is that I've got a lot of respect for Kent Beck. The thing that grabbed me about this talk though was that he was advocating that we could use pull and flow to accelerate learning in the context of an internet startup. The learning is identifying where the value is. What is it that customers are willing to pay for, and in turn will mean that your startup will become financially successful.

I'm a forty something, and I remember when none of this technology existed. I remember when it all was a promise. The microchip was going to change our lives for the better, by giving us all lots of leisure time to spend as we like. Well it didn't quite work out that way did it? Instead technology has made our lives faster and less secure. So given this back drop who is going to willingly turn to technology as a fun thing to do?

Sure people will use technology to interact with other people. I guess that's exactly what I'm doing now, but few will be interested in technology for its own sake. There appears to be a dilemma here. People enjoy interacting with other people, yet increasingly they find that they need to put up with irksome technology in order to do so. The village pub were everyone knows your name is now a thing of the past. Instead we're all sat at home by ourselves trying to make friends through blogs and chat rooms.

Is this a trade that anyone really wants? I doubt it. I'd much prefer being in a real social setting right now, getting this all off my chest :) So the trick it seems is to find ways that are less irksome then others that allow us to interact with each other through these horrible technology mediums we've invented.

The village pub closed down years ago by the way, and even if it hadn't your pals haven't the energy to come out for a drink, because the computer at work means that they now have to do three peoples jobs, so in the evenings they barely have the energy to sit through a TV dinner before rolling into bed :(

Kent is pretty open and honest about his failures in startup land. I think he is being slightly unfair to himself though. No one wants this stuff. People will put up with it only if they really have to, or if they see a way of making money out of it. Technology in the work place is seldom a win win proposition. Often a minority gain whilst the majority loose. Is it any wonder that technology as a leisure pursuit is a hard sell?

The whole drive for being "Lean" and efficient leaves me a bit could. We seem to think that by speeding everything up we can some how make the world a better place. I like the idea that many things of value can't be measured and many of the things we can measure have little value. Try putting a number on the value of your humanity?

I can't help but think that all this technology is being pushed on us all. Kent is right, we can learn which aspects of the technology are less irksome to users, to the point that people will actually put up with it, in order to make contact with their fellow man. But is this really market pull? Or just a desperate means of pushing stuff onto people that deep down none of us really want?

An After thought:

Despite this rant I must admit that I'm still a bit of a techno-geek at heart :) I noticed myself salivating over a colleagues iPad the other day, so I'm still attached to the dark side :) What I'm talking abut here is really a sense of balance and purpose. People aren't machines, and their desires and needs have been built up over thousands of years of evolution long before technology existed. We all need balance in our lives in order to gain a sense of well-being, and our current technology driven society isn't providing that I don't think, which brings me to purpose. Technology is just a tool, its what we do with it that counts. Since the microchip revolution I don't believe we have given sufficient thought to the purpose we should apply all these cheap CPU cycles. Quicker and faster, more with less, isn't a forfilling answer, especially when it ends up turning our lives into one big frantic blur. I guess that's why the iPad is so attractive. It offers a glimpse of a technological future were the medium fades into the background and the content comes to the fore. Alan Kay's Dynabook is with us. Now what was Alan's purpose for the Dynabook again? Oh yes, learning and enlightenment. Lofty ideals, but perhaps we'll get there some day :)

Popularity and Success are not the same thing

2009-05-10T12:12:00.000-07:00

After my last post, I went back and read through Giles blog post again. I missed his main point, and I dismissed what he had to say far too quickly. This video of DHH and Tim ferri, drove the point home for me and is well worth watching. This talk didn't prove as half as successful at Rails Conf, as Bob Martins.

Whilst I still think that Giles point is tangential to what Uncle Bob had to say, I think it is a profound one. That is, that any proclamation that requires free thought is likely to be unpopular. This lack of popularity doesn't equate to failure however, it just means that such ideas are more likely to benefit the few, rather then the many.

Uncles Bob's use of Smalltalk as the epitome of failure, rather cleverly raises the spector of fear, and as we all know fear is the great motivator of the masses. So if as a Ruby programmer, you want to be in a job for the foreseeable future, then make sure and do TDD. It's kind of like a Mom using threats to make sure that the kids eat their greens.

Whilst DHH and Tim and Paul Graham, are fundamentally right, they don't address the psychic of the average mainstream programmer, which is mostly dominated by the fear of not finding a job. So in a way this brings me back to were I started. Sometimes being right isn't enough. Sometimes you need to be effective too, and whilst fear is the worst of motivators, Uncle Bob is addressing the world as it appears to the vast populous, which is an effective tactic.

The world that is Giles reality is a rarefied place and the few people and languages that live there are likely to remain obscure for some time to come yet. The last interesting point is that it appears that DHH has out grown the community he created. This is hardly surprising. Like I say followers don't think for themselves, they rely on others, and as Rails becomes ever more popular, the community will become dominated by followers, just like what happened with Java.

For someone like David, this influx of non-thinkers must be nauseating.

Paul.

What Killed Smalltalk: Sometimes Balls can be Useful

2009-05-10T07:25:00.000-07:00

Uncle Bob, has been his usual charismatic and entertaining self in his keynote speech this year at Rails Conf. He presented a dire warning for the Ruby Community: "What Killed Smalltalk Could Kill Ruby, too". So what is this beast of death? Well Uncle Bob blames the ease at which programmers could create a mess in Smalltalk as the thing that killed off the language, quoting no other then Ward Cunningham to back up his argument. Interestingly he makes only brief mention of another VM based OO language with garbage collection that appeared in 1995, and was market mercilessly and given away for free, whilst Smalltalk vendors were still charging £3K plus per pop for Smalltalk licenses (yes that's over £3000 per seat). Hmm...

Anyway as you would expect, the Smalltalk community is up in arms. James Robertson has this post. And Giles Bowkett makes this response entitled "What Killed Smalltalk: My Balls". Giles response, demonstrates IMHO complete ambivalence to what is actually happening in mainstream programming today. Whilst IMHO Uncle Bob clearly has his finger on the pulse. I have never been a full blown member of the Smalltalk community. I've always been one of those people on the side lines, using Smalltalk for fun and hoping to get a Job that would allow me to write Smalltalk commercially, but never being fortunate enough to do so. I think that gives me a more balanced perspective. I am neither a curly bracket or square bracket zealot, having lived most of my programming career with both. Whilst I've always admired the Smalltalk language, I have noticed that the Smalltalk community has a tendency to be insular and inward looking (with no sign of those balls Giles speaks of). Anyway, here is my comment in response to James Robertsons blog post. I thought I'd post it here (with some minor edits), since I know that very few people take the time to look at Smalltalk blogs:

There are two audiences here. One audience are the 20 somethings who were merely a lusty glint in their fathers eye when the clash of C++ and Smalltalk was taking place in the 1980's. The other audience are the people who where actually there.

For the 20 somethings, I think there is a lot in this presentation to get excited about. First of all it makes the point that good ideas can fade. It also makes the point that great languages provide the programmer with great power. With this power comes responsibility. As developers we have a choice. We can allow ourselves to be dumbed down and given dull tools so we won't hurt ourselves, or we can become responsible and disciplined to the extent that we can be trusted to use sharp knives. This message is an important one. The other great message, Smalltalk, which is now being discussed amongst a new young audience. If any of these young kids are half as curious as I was at their age, then we should expect downloads of Squeak to spike this weekend...

As for the other audience. The people that were actually there. It is easy to dismiss this presentation as a pile of BS. Well it mostly is, but to engage with it only on a factual level is to miss the point. This presentation is not for us. Bob Martin is not a Smalltalk programmer and never was. I was on one of his OOD courses in the 1990's and he asked who in here as used Smalltalk? I was the only person to sheepishly raised my hand. He then went one to castigate Smalltalk, saying that you couldn't use it for anything serious because it would lead to a mess once you got to more then 10K lines of code. Back in them days Bob was a curly bracket language zealot.

I'm glad that James has admitted that that back in the 90's the Smalltalk community suffered from arrogance too. I read Giles response and it sounded more like a tirade. The mainstream developer community at the moment is in flux. They trusted in vendors only to be let down. Now they are looking for new leaders. Most of them are followers, if they weren't they would have self educated themselves long ago, and Uncle Bob wouldn't be able to get away with his historical inaccuracies. The Smalltalk community are in a great position to provide leadership. The Agile movement came from Smalltalk, TDD from Smalltalk, Ruby from Smalltalk...

So why isn't the Smalltalk community doing so? Why wasn't a Smalltalk programmer giving this keynote instead? Where is Alan Kay when you need him or Dan Ingalls? Is it because we've all got a reputation of being grumpy old men? It is up to the Smalltalk community to make themselves relevant to these new young kids. Bob is doing his best to tell a story second hand. The Smalltalk community should be telling this story itself. Looking in the mirror at yourself and your own short comings, is much harder then pointing fingers at others.

Paul.

The road to mastery - Practice, Practice, then Practice some more...

2009-03-15T10:45:00.000-07:00

The craftsmanship movement has hit the ground running. First a conference in London organised by Jason Gorman. Looking through Jason's Blog, and from the video, it looks like the whole Xtc London crowd were in attendance. Jason is an established member of the Xtc crowd, so I suppose it shouldn't have been surprising. Familiar faces like Ivan Moore and Nat Pryce. It brings me back to my early XP days, when I too attended such conferences and the excitement and delight of it all, not to mention the friendships and the insights. What fond memories.

So after going full circle, the "Agile" movement are now back were they started: Writing Code! Or should I call it the post Agile movement? Or better still lets just call them the folks who like to program, get stuff done, and take a pride in doing it well.

Micah Martin was there, and presented his Langston's Ant Ruby Kata. Now Micah's dad, Uncle bob is the guy responsible for teaching me the fundamentals of OO design. So anyone schooled by Uncle Bob, will probably have something to say that would prick my interest. Watch the video. Its a real expose of the Ruby language and the mechanics of programming in general.

Micah borrowed the word Kata from marshal arts. A Kata is an almost ritualised performance of a set of martial art skills, with the desired aim of re-enforcing those skills through practice. So Kata's are meant to be performed again and again, until the practitioner becomes use to the moves involved. Its a wonderful idea, and is synergistic with Ron Jefferies emphasis on practice and learning through doing. Ron's bowling game example could be considered yet another Kata. Hopefully Micah will develop this idea further and I'm looking forward to a whole catalogue of Kata's that the aspiring programmer can learn and practice. Katas to me seem like a great tool when going through the Shu (hold) phase of learning.

Focusing on Micahs Ruby Kata, what was interesting to me was his interpretation of BDD. I have been doing quite a lot of BDD lately, and I have found the style of BDD I've been learning a tad verbose. Micahs BDD style using Rspec is very similar to my own TDD style, and a lot more appealing as a consequence. Given that Uncle Bob learned TDD from Kent Beck, and I learned TDD from a bunch of XPers (Ivan Moore, Steve Freeman, etc) who presumably picked it up from Kent Beck too, its clear to me that Micah and I share a common BDD/TDD lineage. This raises an interesting point about style.

Micahs software shop is called 8th Light, and they conduct an apprenticeship program. Now I always thought of an apprenticeship as something for novices, but 8th Light take a different slant on the apprenticeship idea. Rather then an apprenticeship being just a means of teaching someone the basics, 8th Light also sees an apprenticeship as a means of teaching someone the 8th Light Style. This is another great idea. From my waterfall days, I have known that great programmers will get the job done, using what ever approach is familiar to them, and there is no one right way, No Best Practice, No one style.

Corey Haines explores the idea of style with Paul Pagel of 8th Light in this video. They also talk a bit about the recent Craftsmanship Manifesto and what motivated it. They suggest that programmers should learn more than one style during their Journeyman phase. Again this is something that is synergistic with my experience. I remember working with Ivan Moore and Erik Deorenenberg on the same project. Both great programmers, but each with a different style. Whilst I found Ivans "Smalltalk/Python" biased style more appealing, Eriks prolific nature and practical bias was extremely informative too, and I vowed to get my keyboard skills up to Eriks levels (which is still something on my to do list :)).

So the Journeyman must be prepared to break with his master and explore alternative styles. This seems to me like the Ha (break) phase of learning. At 8th Light they are hoping to facilitate this by organising exchange programs with other software shops.

So there we have it. People walking the talk, and software as a profession actually being practiced. It is interesting contrasting the Craftsmanship movement with the Kanban movement, both recent off shoots of Agile. One group (Kanban) are busy telling others how to manage software development, and are selling their services as Consultants, whilst the other group (Craftsman) are busy writing software themselves for real customers and sharing their practices and insights for free with anyone interested in learning.

I saw this on Jasons blog:

Q: How do you get to Carnegie Hall?

A. (craftsman): Practice, practice, practice.

A. (consultant): Pay a respected market research company shedloads of money to write a report that concludes you're already at Carnegie Hall, and then make a killing selling maps and directions (even though you've never actually been there)

I think it says it all. All's left for me to do now is to get practicing myself. I'll think I'll start with the Langston's Ant Kata in Newspeak. Now that should be fun!

Post Agile - Beyond Best Practice and towards Professionalism

2009-02-14T15:00:00.000-08:00

The Agile community has been going through growing pains of late. InfoQ have been tracking this in a number of good articles referencing blogs posts by various members of the Agile Community.

I'm not going to reference all the blogs, but I think this introspection is a good thing, and I sense the birth of something new as a consequence. Bob Martin describes the ethics and the ethos that underlies this new thing in this presentation. He talks about a consensus arising around favoured software development practices, and he raises the notion of an accepted standard of Craftsmanship, which when tied with the ethical issue of actually delivering to our customers what they payed for, raises the possibility of software development actually becoming a profession. The idea of craftsmanship is further explored in this blog. I like the idea expressed here of "doing it right" rather then merely "getting it done". Andrew Walker talks to the ethical issue by making the point that we can't really in any honesty call ourselves a profession whilst the failure of software projects is still endemic costing our customers millions for no or little return.

Jim Shore has been busy blogging about the future of Agile also, the most infamous is this post which shines light on the problems that arise when people adopt Agile Management practices like Scrum, but choose to skip the more difficult challenge of adopting sound technical practices. The post by Jim I find the most revealing though is this one on Kanban. The reason why I find this post interesting is that it raises the question of what is "Best Practice".

Jim tries to explain why people are choosing to eschew iterations, calling planning meetings waste, and are choosing to shrink their backlog, calling it excess "inventory". You will recognise "kanban", "waste" and "inventory" as terms bored from Lean Manufacturing. Lean ideas are all the fashion at the moment it seems :). Besides the fact that these ideas are being applied out of context, labelling them "Kanban" (which interestingly is misappropriated since the term already has a specific meaning), reveals our desire to label and promote "Best Practice". In the comments Steve Freeman makes this point, and says that after speaking to David Anderson, one of the original people who invented what has become known as "Kanban", he found out that the reason why he did "Kanban" had nothing to do with eliminating waste, but rather it was a pragmatic response, to an incumbent waterfall culture that didn't allow for iterations. David himself later comments and confirms that Steves summary is accurate.

So one persons chosen practice in a given context, is marketed as best practice for a whole industry. Some in the Agile community have been clear about the need for self organisation and doing what works for you, but historically process improvement has always been viewed in the western world as the adoption and the enforcement of "best practices", and old habits die hard. As a consequence, selling "Best Practice" is now an established cultural phenomena in itself. The Agile Community as been as guilty of (mis-)selling bottled snake oil as all the other process improvement fads that have come and gone before it, despite the small print in the Agile Manifesto that speaks to the contrary. This opportunistic selling perhaps explains why the Agile banner lacks credibility in the eyes of many. They've seen it all before. Ethical issues aside, our culture of promoting "Best Practice" is very naive. Process improvement for an activity as complex, difficult and diverse as producing software, cannot readily be bottled and sold; which probably explains why we still aren't a true profession today.

A more recent article on InfoQ refers to a blog post that makes the point that Agile and Lean aren't the same thing. Now this is something that should be self-evident to anyone who as taken the time to read the work of Taiichi Ohno on the Toyota Production System for themselves, and shouldn't need explaining. Even so, I think that recognition that we cannot in all honesty promote Agile, Lean, Kanban or anything else as "Best Practice", devoid of context is a big step forward. The emerging consensus is that what really matters is developing competence in a useful toolbox of favoured practices, leading to a high standard of craftsmanship. This is not news, but it is nice to see this common sense point of view finally getting the top billing it deserves. It also perhaps signals that as an industry we are maturing, and now able to face the difficult questions that arise when you ask what does it take to be reliably successful at producing software. This fundamental question has hung in the air, ever since Fred Brooks first raised it. I'm tempted to go further and suggest that this change in the mood music may also be signaling the embryonic beginnings of professionalism. Or perhaps that would be going too far :)

What are your thoughts?

I don't know

2009-01-02T15:52:00.000-08:00

If you read something new that is contrary to your current understanding you are faced with several possibilities:

The new idea stems from a new context which you haven't experienced. In this new context the old idea that you are familiar with no longer applies.
The new idea hasn't been rigorously tested, hence the need to seek corroborating testimonies.
The new idea has been corroborated, but the corroborators are involved in a conspiracy of lies in an attempt to deceive.

Now before you jump to option 3, you would rigorously explore options 1 and 2 first, for the simple reason that you may learn something new. Well at least you would think so.

In my experience this is seldom the case with programmers. In most instances when faced with a new idea that challenges current "best practice", I see programmers jumping to option 3.

This blog post and the comments that follow is a clear example.

So why is this? I think it has to do with how programmers process information. As left brained people, they build ever more elaborate models of understanding of what is "best". New information to the contrary challenges that model and rather then see the model fall like a deck of cards there is a strong impulse to reject the new idea.

I have blogged about this before:

Not knowing shouldn't be threatening, a state that should be extinguished quickly, with ever more elaborate models. No, not knowing is just part of the rich fabric of life. In fact there is power in the unknown. In acknowledging that there isn't a simple formula, order can be found in chaos.

Finding order in chaos is one of the most basic tenets of Agile Development. It is this philosophy that eschews any attempt to be deterministic and know in advance what is best. Instead Agile development embraces an empirical approach of trail and error, suck it and see, where not knowing and discovery is a natural part of the process.

This philosophy seems to be a hard sell. Everyone wants a fixed model, kind of like painting by numbers, which you can buy off the shelf and will solve your problem each and every time. In an attempt to fix the model scepticism is deemed as negative. The vice of non-believers who refuse to "get on message" about what is "best".

Keith Braithwaite takes a contrary point of view. Rather then seeing scepticism as always bad, he believes that healthy scepticism is a good thing, a sign that you are actually thinking and processing new ideas on their own merits. I agree. The problem with 'best practice" and a "fixed model" is that for any non-trivial process "best" is always conditional. So in different contexts what is best will differ. So what is right for one team may not be right for you. So there is no ready formula, and you need to discover what is best for each specific context, which means admitting that perhaps you don't know. This is why you can't master Agile development by just reading a book, and most Agile advocates suggest that you commandeer the help of an experienced coach who has experienced several different contexts to help guide you in discovering what is right for you.

Keith makes his point very well in an interview on InfoQ. At around 24 minutes and 49 seconds in to the interview, the interviewer asks "How do you choose the right Agile management tool if you are new to Agile?". To which Keith answers: "I don't know". From this admission you can actually see a process of discovery ensue. Keith then goes on to say that in lieu of knowing he would suggest using the simplest thing that works, an Excel spreadsheet. After using the spreadsheet for a while and writing macros to automate the calculations you needed you would learn what it is your process needed from an Agile project management tool and would then be able to decide which tool if any on the market best suited you.

It is a magical moment, with Keith shacking his finger in excitement as he discovers the answer to a question which he previously didn't have an answer to. The interesting thing for me is that the discovery started off with the statement "I don't know".

How many of us would respond to our boss when asked a question like "What tool should we use?" with "I don't know"? Very few in my experience. What is more likely is a long diatribe based on our prior context and what the book says or what we've read in reviews. As a consequence the opportunity for discovery is lost.

Take the time to watch Keith's presentation and try saying "I don't know" more often. You may find that it accelerates your learning tremendously.

Further Thoughts:

Alan Kay has another explanation for why people with prior context find it difficult to accept new ideas:

"Kay blames this lack of innovation on the fact that most adults employ instrumental reasoning to evaluate and apply new ideas. This means that adults have difficulty evaluating new ideas because they're carrying too many existing goals, and too much context to be able to see the full potential of new ideas. One way to overcome this problem is to work with children. Children won't know that something is difficult or that hundreds of papers have been published explaining the intricacies of some particular problem. They are more likely to experiment where adults would have given up, and to ask, "So what else can I do with this?"

I think this is similar to my "preserve the existing model" idea, where people reject new ideas that don't fit their established understanding. The same issue is also seen in science where breakthroughs happen during moments when some one is brave enough to go against the established orthodoxy. It is interesting that Alan Kay believes that children don't suffer from this. This fits with the proverb "you can't teach an old dog new tricks". So one solution is to try and maintain a child like curiosity and always ask the question Why? in the annoying way children do. I do this and it works well. If you ask why 5 times, it is usually enough to reveal something new.

Craftsmanship

2008-12-18T16:18:00.000-08:00

Simon Brown has been re-evaluating the role of Software Architecture in response to the question:

Why do we need this new software architecture stuff? We've managed fine until now and projects *have* been successful.

Good question. Some teams don't dwell on the subject at all, any yet have produced remarkably well designed systems that have stood the test of time. Unix (Thompson and Richie), BSD Unix (Bill Joy) and Linux (Linus Torvald), just to name a few. In the case of Linux, there wasn't an opportunity to perform Big Architecture Up Front (BAUF) as a specific activity, since it was the result of the collaboration of literally thousands of developers on the web, so the Linux architecture we see today is truly an emergent entity devoid of a master plan.

What is architecture anyway? Simon concludes:

I say that architecture is about introducing structure and guidelines to a software project, and I've come to realise just how true this is. Without formalising these good ideas, a codebase with some otherwise great ideas can appear inconsistent, with the overall system lacking coherence and clarity.

In the discussion that follows, Simon goes on to explain that the architecture is "the big picture". Although he avoids using the word "design", I interpret what he says as architecture is the highest level system software design, which with a lot of programming languages is something that is difficult to gleam from just the code. I agree. What is disturbing however is that he flirts with the idea that defining this design is some how a different activity from programming, and calls for a dedicated role with dedicated skills. So architecture and consistency is something to be imposed onto a team of developers who can only code at the detail level and require an higher thinker, the architect, to impose "structure".

I like Simon, and I think he senses the irony in this. The team can't think and can only code, so the Architect must do their thinking for them. I wonder where this mindset stems from?

We will win and you will lose. You cannot do anything because your failure is an internal disease. Your companies are based on Taylor’s principles. Worse, your heads are Taylorized too. You firmly believe that sound management means executives on the one side and workers on the other, on the one side men who think and on the other side men who only work.”
Konusuke Matsushita – Panasonic

Simon can't be blamed. The values in our industry are such, that in many instances this is the only politically acceptable way to address the problem. People who are only paid to code and not think will struggle with architectural concerns and do need help. I pointed out that these same people will struggle with other aspects of programming too like listening to customers, effective collaboration and testing for example. Andrew Walker goes on to make the point quite eloquently that programming and architecting are one activity are can't easily be separated. I agree. In fact as I become better at my craft I no longer do BAUF (I once was an Architect in a past life) and I now allow my architecture to emerge as I write code.

So de-skilling the developer role by splitting out development activities like designing, communicating, listening etc into separate roles runs counter to the idea of good craftsmanship IMO. Craftsmanship is holistic. So how did we get here? Well de-skilling the workforce does allow you to pay them less and hire and fire people at will. It also puts those pesky all powerful "programmers" in their place, and may serve to provide the management with the illusion of control. There is a whole industry out there selling "Silver Bullets", and suggesting that software development can some how be automated and made simple by the latest Framework or piece of Middleware. And finally there are a bunch of ancillary industries like Business Analysts, Systems Analysts, Enterprise Architects, Solution Architects, etc, willing to do all "the thinking" at a fee, leaving the bulk of the team to only work.

Henry Ford had huge success with this approach in the 1930s, drastically reducing the cost of motor car manufacture by employing cheap unskilled immigrant labour. In recent decades however, in the face of competition from companies whose workers are empowered to think for themselves this Taylorist management approach has been found wanting. To see evidence of this you only need to open your newspaper. The Big 3 US Motor Car Manufacturers have recently gone cap in hand to congress for a bail-out. Toyota, Nissan etc build and sell cars in the US too, but they aren't asking for a bail-out! These Japanese manufactures are Agile enough to adapt to the needs of the market and are riding out the current economic storm. Toyota are even recruiting in the US for a new plant in California I believe. So why are these Japanese companies prospering at the expense of Ford, GM and Chrysler? In the 1950's Japanese motor manufacturers took a look at Ford, GM etc and rejected their Taylorist approach in favor of the ideas of Edward Deming. Deming felt that "scientific management" wasn't enough because it undervalued people and ignored human psychology. Could the Japanese choice of Deming over Taylor have something to do with the value they place on people?

Value and respect. I work as a Coach and I would never be as arrogant to suggest that I am the only person on the team qualified to determine the "big picture". I would like to believe that I have mastered my craft, but does that mean that I am the only person qualified to think? No this is arrogant nonsense and is exactly what Konusuke is speaking about.

Every human being can think and each of us should be encouraged to think. Our brain is our most valuable asset. The issue is that we aren't born knowing how to program well, and we all need to learn (even those of use who later become architects :)). So collaboration, architecting, coding, listening, designing are all things that must be learnt. The Japanese speak of Shu-Ha-Ri (Knowledge, understanding and skill), as the three phases of learning. A computer science degree will only provide you with some basic knowledge. To understand you will need to practice, and to gain skill will require repeated application in varying contexts. And of course you can't learn on your own. To learn effectively you need a teacher.

In past times when a man wanted to learn a craft he became an apprentice, and would live and work alongside a master craftsman for several years, receiving only food and lodgings for his efforts. After years as an apprentice he was deemed qualified and would become a journeyman, allowed to charge for his labours. A journeyman would then move from workshop to workshop gaining experience as he went. The best journeyman could become master craftsman themselves, setting up their own workshop and taking on apprentices of their own. Along the way people not suited to the craft would drop out rather than waste time, ensuring that standards remain high, and people found crafts that they were suited to. So in the past the west had its own version of Shu-Ha-Ri it seems. In fact this tradition still lives on in professions, like Lawyers, Doctors, Architects (the building kind), Fashion designers etc. So it looks as though the software Industry chose an unfortunate template in the Manufacturing industries to model itself on.

I can imagine that Henry Ford, must have seen traditional Coach Building Master Craftsman as an extravagant waste of money and a threat to his profits. Surely the job could be split into a number of simple unskilled roles with much of the work automated. This proved to be true (for a while) for motor car manufacture, but the same approach has failed us when applied to software. Why? because software development is not manufacture, it is not repetitive, instead it is creative. It is new product development, akin to fashion design, car design etc.

Successful software development calls for master craftsman like the names I mentioned earlier: Bill Joy, Linus Torvald etc. These masters have a responsibility to pass on their skills to apprentices whose job it is to think and learn by example at the knee of their master. The Agile community have gone back to basics and understand that developing people is the only way to pull the software industry out of habitual failure. People over process and tools.

Infoq has been tracking the tour of a Journeyman as he visits various workshops in an attempt to learn from the masters. His blog posts make interesting reading. Especially the way the learning occurs: side-by-side at the keyboard whilst programming. The master and the journeyman discussing the work and lessons being learned on both sides. An age old tradition, you can imagine Michelangelo working with his apprentices and journeyman in the same way.

Looking at the videos and listening to the interviews it seems such a natural way to learn, and the ideal solution for teams that are struggling with architecture or anything else. Hire the best people you can afford, and have them mentor their colleagues in a learning environment.

So obvious, yet Kevin a colleague of Simon seems to have missed this possibility. The idea that programmers can be helped to think for themselves escapes him as a possible alternative to the Architect going in and doing the thinking for them.

Perhaps Konusuke Matsushita is right. We do have an internal disease.

Object 101 - What is an Object?

2008-12-02T05:48:00.000-08:00

Judging from the responses to my last post on uniformity it looks as though I started off with the bar too high. So lets lower it a bit. Lets go right back to basics and try and define what I mean by Object.

Concept

Firstly, the idea of Objects is a conceptual one. Alan Kay and his team did research for over a decade trying to understand how best to structure computer programs and settled on the idea of Objects. The concept of Objects can be explained by taking a biological analogy. Alan Kay speaks of the idea of an identifiable cell, which has a membrane encapsulating its insides from the outside world. Each cell is autonomous and goes about its job independently from other cells, but cells do collaborate in a loosely coupled way by sending (chemical?) messages to each other.

So Objects are analogous to cells and the key defining characteristics of an object are identity, encapsulation and messaging. Notice I haven't mentioned classes or inheritance. These things are merely just one approach to implementing objects and defining what goes on inside the membrane. There are object orientated languages like Self which eschew classes all together.

Lets explore these fundamental characteristics in more detail:

Encapsulation

The power of encapsulation is that it hides the implementation from the outside world. In the same way that a cell membrane hides the inside of a cell. The outside world need only know the message interface of an object (known in Smalltalk speak as the message protocol). So when I send a message to an object, I have no idea how that object will process that message. The object is free to process the message in anyway it likes. This leads to the idea of polymorphism, which is a Greek word meaning "many forms". Since an objects implementation is encapsulated behind its message interface, the implementation can take any form it likes. This means that message implementations are free to perform any side effect they wish as long as they satisfy the message interface. Notice again I have not mentioned classes or subclassing. Again subclassing is just one approach to achieving polymorphism. Any object that satisfies the message protocol is viewed as a polymorphic variant, and can substitute any other object that shares the same protocol.

Messaging

During their research Alan's team looked at a number of ways of getting their objects to communicate with each other in a loosely coupled fashion. If you take the biological analogy to its extreme, then each object should be totally autonomous and share nothing with the others. This means that each object should have its own cpu, its own memory, it own code, Alan Kay as even argued that each object could/should have its own IP address as a global identity. So an object in this view of the world is a networked computer. OK how to hook these objects together. Well the natural solution is asynchronous messaging, just like you get with e-mail. Since each object has its own cpu it doesn't want to block and wait until the receiving object processes the message. So an object can send a message without blocking and the receiver will send back an answer into the objects inbox its own good time. This approach is what we call the Actor model today, and as someone kindly pointed out, Alan Kay first explored this approach in an early Smalltalk back in the 1970's. Interestingly the Actor model has come back en vogue with Erlang which has adopted it to implement "share nothing" concurrency. This is why some people say that Erlang is Object orientated.

The share nothing approach to objects could be considered to be a bit inefficient. I'm not sure of the history, but Alan Kay and his team decided to move on from asynchronous messaging to a more restricted synchronous approach. With synchronous messaging all objects share a common processor (or thread) and message sends block until the receiving object has completed processing the message and has responded with an answer. This is the messaging approach that was settled on in Smalltalk-80 and released to the world in 1983.

State and Behaviour

So we now have objects sharing cpu, but each object still encapsulating its own memory (program state) and its own code (behaviour). The state and behaviour of an object is private (encapsulated). So what happens when two objects share common behaviour, but have different state? As an example what if I have two bouncing balls, one red and one blue? Do I implement two objects separately, duplicating the code? Obviously there is an opportunity here for these two objects to share common code.

As part of the private implementation of these two objects, sharing code is desirable ( the DRY principle). One approach is to create a new object to encapsulate the common behaviour. In Self they call this a trait object. This leaves the two original objects just containing state (red for one and blue for the other). Common messages sent to the red ball and the blue ball (like the message 'bounce') are delegated to the shared trait object (through something called a parent slot) which encapsulates the common code. The red ball however may decide that in addition to bouncing, it can blink too. Blink meaning changing colour from red to white and back again repeatedly. So in addition to 'bounce' the red ball adds the message 'blink' to its message interface. This behaviour is not shared with the blue ball, so the red ball will need to have its own blink code which is not shared. So in Self objects can choose to share some behaviour or may choose not to share any behaviour at all.

In Smalltalk-80, the idea of not sharing implementation was relaxed, although conceptually the idea is still useful. So in Smalltalk-80 all objects share behaviour with other objects of the same kind, leading to the idea of classes of objects. Again I'm not sure of the history, but I believe this is a more efficient approach then the approach adopted by Self (Self came about in the 1990s many years after Smalltalk when memory was cheaper and CPUs faster). So in Smalltalk all objects share common behaviour with objects of the same kind through a common Class object.

Classes

So we finally get to the idea of a Class. A class is merely an implementation convenience, and unlike what the C++ proponents would have you think, the idea of class is not central to OO. In the same way that objects can share behaviour through a common class object, so can class objects share behaviour through a common superclass, leading to the class hierarchies we are all familiar with and tend to associate with OO programming.

Notice I use the term Class object. In Smalltalk a class is a factory object. Incidentally a lot of the so called OO patterns in the Gang of four book are really C++ patterns. So for example there is no factory object pattern needed in Smalltalk where you get factory objects for free.

A factory object is an object that creates other objects. In Smalltalk such objects are stored in a global hashmap called Smalltalk and are available throughout your program. Global objects in Smalltalk are identified with a capitalised first letter in their name by convention. Classes in Smalltalk are just global factory objects and hence all have a capital first letter. This convention has been carried forward into C++ and Java.

So how do you create a class? Well you send a message to another class you wish to subclass. It will come as no surprise to know that the message is called 'subclass'. This message answers a new class. To this class you can add class methods and instance methods again by sending messages. Instance methods are methods that you want to appear on objects that are created by this class (remember a class is a factory), class methods are methods that belong to this class itself and defines the class's behaviour. Also you may want to add class and instance variables to your class to encapsulate state.

So how do you create an object? Well you send a message to the factory responsible for generating the kind of object you want. This means that you send the message 'new' to the class object.

This is where languages like Java and C++ take a cop out. 'new' in such languages is not a message send on an object, instead it is a keyword in the language. This means that you cannot override new and hence the need for the gang of four 'factory object pattern' in these languages.

Back to Smalltalk. In response to the message 'new 'the class will answer a new instance object. So now we have a new object. Incidentally we skipped over how objects are created in Self. In Self any object can act as a factory and is able to create a copy of itself. So in Self you send the message 'copy' to the object you want to copy. The copied object is now a prototypical instance which is why Self is called a prototype based OO language (rather than a class based OO language).

MetaClass

Back again to Smalltalk. We have factory objects (classes), and we have instance objects (objects), but where do the class methods live? Where for example is the code for 'new'? As I said before a class has two roles, one to define it own behaviour (such as defining new) and the other to define the behaviour of its instance objects. I called its own behaviour class methods. This behaviour belongs in another object, the classes class (classes have a class too). I think we need an example:

aTranscriptStream := TrancriptStream new.

The 'new' message implementation is defined in the class of the TranscriptStream Class object. The class of 'TranscriptStream' is called 'TranscriptStream class' which is also known as the meta-class. 'TranscriptStream class' also has a class: 'TranscriptStream class class', and so it continues. 'TranscriptStream class' and 'TranscriptStream class class' etc are all implemented as the same object, the Meta-class object (otherwise we could go on for ever). This circularity is one of the beauties of Smalltalk. Meta-classes do not exist in C++ and Java, and hence why 'new' is a keyword in these languages.

So the 'new' message is sent to the TranscriptStream object (which is a class) and the implementation is defined in the 'TranscriptStream class' object (which is a meta-class). Now how do we end up with 'Transcript'? Remember that in Smalltalk globals all start with a capital letter. To make something global I need to add it to a global hashmap called Smalltalk along with a global identifier (symbol):

Smalltalk at: #Transcript put: aTranscriptStream.

Then I can do this:

Transcript show: '3 is less than 4'.

Uniformity

The message 'show:' in the previous example is sent to the Transcript object (which is an instance) whose implementation is defined in the TranscriptStream object (which is a class). Interestingly you cannot tell whether Transcript is a global instance or a class. I initially mistook it for a class in my discussion with Stephan until I looked it up. The thing is with Smalltalk is that it doesn't matter. Instances, classes and meta-classes are all the same thing, they are all objects. Smalltalk is uniform and everything is an object, which is where I started.

Updated 4/12
Replaced Transcripter with TranscriptStream. Instances of both these classes satisfy the 'Transcript' protocol, but in Squeak Smalltalk the global 'Transcript' object is an instance of TranscriptStream. I found this out by printing 'Transcript class' in a workspace. Another approach is to print 'Smalltalk at: #Transcript'.

Objects 101 - Uniformity

2008-11-30T17:54:00.000-08:00

Some interesting comments to my last post have prompted my to expand more on what I believe is good OO.

For me getting to know good OO started out with being skeptical about Bad OO and saying "I don't get it". This is why I think I understand Joe. Healthy scepticism is a good thing, especially when something blatantly just doesn't add up. I could go into a rant about why programmers find it difficult to say "I don't know" or "I just don't get this" and blindly admire the “kings new cloths” even when the king is naked, but I'll leave that for another day :)

As an example of good OO let me reproduce the Smalltalk code example from my response to a comment by Paul Homer to my last post:


 3 < 4 ifTrue:[ Transcript show: ‘3 is less than 4’].

Ok lets try and express this in a more familiar OO language, Java:


if ( 3 < 4) System.out.println(“3 is less than four”);

What is bad about this? Like Paul Homer pointed out the instructions (if, <, ..) clearly take precedence over the data (3, 4, ..) in true procedural style. Also there is only one object here "System.out", which given that it is a global static object is hardly an object at all. System.out.println() is no different then ( #include <system/io>) printf(). So the whole statement is not OO, it is procedural and would look pretty much the same written in C.

But Java is supposedly an OO language, so surely I can express this as objects. Lets try:


(new Integer(3)).isLessThan(new Integer(4)).isTrue(new Block {
         void execute() {
             System.out.println("3 is less than four");
         }});

Ok. I've created a DSL here in Java to get rid of the primitives and procedures and express everything uniformly as objects. So what is so bad about this? Well it doesn't read half as well as the Smalltalk example. All those parenthesis and periods tend to obfuscate the intent. Secondly I would need to write my own Integer and Boolean classes, overwriting the ones in the Java standard library. Java's Integer doesn't understand the message 'isLessThan" and Java's Boolean object doesn't understand the message "isTrue". Also the use of an anonymous inner class to simulate a block seems rather verbose, but without closures what else can I do?

So writing pure OO code in Java is difficult to say the least. Does this matter? Well if you are trying to learn OO with an hybrid procedural/OO language, then I think it does. I for one (using C++) definitely found it a challenge.

What do you think?

Paul.

Why Bad OO Sucks

2008-11-27T05:56:00.000-08:00

Anyone who as read my blog knows that I am an OO advocate, but I was watching Joe Armstrong the other day of Erlang fame and he posed the question: "In which Object should you place a given function? To me the choice seems arbitrary". Now if you've got a solid grounding in OO design principles then the answer to this one is easy. Keep the function near the data. So the object with most of the data is where the function should be. I have posed this very same question at interviews and most of the time supposed experienced OO Java developers are clueless.

The fact that a language designer is asking this question in itself is interesting, and says a lot about how OO has been misrepresented over the years. Joes critism is thoughtful. He outlines why he believes OO sucks in a blog post. He then asks the question, why did OO become so popular? I repeat each of his reasons here:

Why is (Bad) OO so popular?

Reason 1 - It was thought to be easy to learn.
Reason 2 - It was thought to make code reuse easier.
Reason 3 - It was hyped.
Reason 4 - It created a new software industry

I agree with all of Joes points.

Reason 1- Bad OO is easy to learn because it requires no learning at all. C with Classes is still C. And a class can be used the same as a struct if you choose. The whole ORM industry is built on using classes as structs :) So a bunch of programmers have transitioned from C to C++ to Java etc without learning almost anything at all.

Reason 2- Code reuse as always been speculation. The languages where object reuse has been achieved are not the popular ones. Take a look at NextStep. They managed to produce a pretty reuseable Financial Framework in the 1990s, but it was all written in Objective-C. The "industry" had decided at the time that C++ should be the OO language of choice and went off and tried to build "Pink", "CommonPoint", "OpenDoc" etc which all failed. Why? Could it be that C++ is a static language and the idea of "live" resuable Objects only works with a dynamic language? Even with dynamic languages reuse is illusive. Why? Well back to the idea of categorisation. No two things in this world are truly identical. So my Banks idea of a Bank Account is different to your Banks. So even with the binding problem solved by using a late bound dynamic language, it doesn't mean I can send my Account Object to your Bank and expect everything to work. Proper OO aficionados know this and for this reason alone do not seek to reuse objects out of context.

Reason 3 - Hyped. No arguments here :)

Reason 4 - OO didn't create an industry. The IT industry created an industry. In fact the father of OO Alan Kay has spent the last twenty years bemoaning what the IT industry has done with his baby. Reasons 3 and 4 go together. The industry hyped OO with promises of reuse etc and then felt obliged to go off and build middleware to deliver on the hype. The astute amongst us noticed pretty early on that hey this stuff is getting pretty complex and heavyweight. The rest merrily jumped on the bandwagon with COM, CORBA, J2EE etc. So as Joe rightly points out the industry created a problem of its own making then went about selling us stuff to solve it.

At this point I should go on to defend good OO. But Joe wasn't speaking about good OO. Good OO doesn't suffer from all these problems and good OO isn't popular. Infact good OO is still in obscurity and suffering from funding problems, something that cannot be said for Bad OO.

No, I have plenty of other posts on this blog that speak to the qualities of good OO. And I am not going to say that good OO is intrinsically better or worst then good FP. I would just say that they are different approaches to modeling, each with its own sweet spot. And as always the thing that really matters is the fleshy blob behind the keyboard :)

Small is Beautiful

2008-11-23T17:40:00.000-08:00

Computers are getting faster and more powerful all the time, yet whilst the average mobile phone as more processing power then all the computers NASA used to put man on the moon combined (well I'm not sure whether this is true, but if it is it wouldn't surprise me:)), our software of today is still no better then Doug Englebarts in the 1950's.

So why are us programmers having such an hard time keeping up with the hardware? The root of the problem is complexity. Whilst hardware has got much faster, it still performs a small number of very simple things like ' load', 'add', 'subtract', 'accumulate', 'shift', 'store', etc. It is the sequence in which these simple things are performed and the speed at which they are executed that gives the illusion of the computer being clever. In actual fact the computer isn't clever at all, the clever part is arranging these simple instructions into a useful sequence, and this is where programmers come in.

So programmers are the clever ones. Yes, but our cleverness is finite. Evolution works at a very slow pace and cannot be expected to keep up with Moores law :) So how can we expect to utilise ever more powerful computers if our brains just can't keep up? Well the starting point is realising that people are indeed the weakest link. Our cognitive abilities are finite and we need to organise our programming activities in such a way to best exploit our inherent strengths and respect our limits as human beings. This leads naturally to the subject of programming language design, but I'm not going to go there today.

No what I want to speak about is chunking and the magical number 7 plus or minus two. The way the theory goes is that us as human-beings can hold around 7 +/- 2 things in our brains at one time. Once the number of parts in a system become larger then this, then we are well advised to abstract and create hierarchies of things, ensuring that the 7 +/- 2 cognitive limit at any given level in our hierarchy is observed.

Anyway, back to software, 7 +/- 2 doesn't only apply to the organisation of computer programs, it also applies to teams. Small software teams tend to be able to adapt to changing requirements in away that large teams just can't. As a strong believer in Agile development and Emergent Design, this ability to turn on a sixpence is invaluable to me, and I tend to avoid large teams.

Experience has proven to me, that when it comes to software development that small is beautiful. After proclaiming this for many a year I have finally stumbled on the original paper that explains the theoretical underpinnings of this point of view:

The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information
by George A. Miller.

Enjoy.

Shu-Ha-Ri (Knowledge, Understanding and Skill)

2008-11-19T08:39:00.000-08:00

InfoQ has picked up on a couple of blog post by Jim Shore recently. One where he speculates that Kanban and Lean approaches are viable alternatives to methodologies like Scrum and XP. Another where he says that Agile is in decline.

All the terms in italics in the above paragraph are just labels. Labels are just a convenient way to categories things. What we all would do well to remember is that in the words of Hayakawa: "the label is not the thing". Just because something is labeled Agile doesn't mean that it is an actual instance of the thing the original advocates had in mind. The Japanese understand this, and have developed a method of learning where they recognise different levels of mastery over the thing that is being taught. Shu-Ha-Ri, or roughly translated: Knowledge, Understanding and Skill. Just because some one knows the label it doesn't mean that they understand the thing. And just because they understand a specific instance of the thing, it doesn't mean that they have the skill to improvise and create a new instance of the same thing in a new context.

Categorising things and applying labels to them is a practice that is full of pitfalls. I will be posting more on this subject in the near future.

America is not Voting

2008-11-01T09:19:00.000-07:00

I came across this on the BBC. Now why on earth in the richest country on the planet are people queuing for hours just to exercise their right to vote? I am living in the US right now and watching the election as an outside spectator is very interesting.

Traditionally around a third of Americans bother to vote. In fact one of the biggest grass roots issues over here has been voter registration. So why the apathy? This time around voter turn out is expected to reach record levels. So it looks like the silent majority have finally found their voice.

Central to the idea of democracy is the responsible citizen making an informed choice. In the UK we take our voting responsibility very seriously indeed. Which is as it should be given that people have died defending our right to vote. The whole ethos of our (free) education system and our public service cultural institutions like the BBC is geared to producing well rounded and well informed citizens able to utilise their vote intelligently. Over the years this has led to consensus politics, where there is broad cross party agreement on a number of major issues. When you have vice presidential candidates who can't tell you which newspapers they read, doesn't that say something about the strength of your democracy? If the politicians aren't informed, what are the chances of the electorate being informed? Isn't this the root cause of the polarisation that is so self evident in US politics today? The informed versus the uninformed rather than left versus right?

I picked up this quote from a documentary I once saw on the plight of the American Indian. An old Indian chief on a reservation was asked what he thought of the white man:

"The white man has many great things, but he cares not whether his people are wise".

Apt words, which still have relevance today. Living here now, I can say that Americans are wonderful people and worthy of the claim to be the greatest nation on earth. With any luck they will make the right choices and begin to address the weaknesses in their democracy and become even greater still.

The defining moment in the election so far for me was Colin Powells contribution. Colin Powells America is a country I would be proud to be a citizen of.

Comprehending Large Software Systems

2008-10-03T01:15:00.000-07:00

William Martinez has started blogging again. He as an interesting blog outlining the roots of emergent design. It is worth a read. One factor that limits peoples abilities to apply emergent design effectively in a team context, is the high degree of design communication needed across the team. Since the design is always changing, maintaining a shared comprehension of the entire system design across the whole team can be a challenge.

The XP Solution
Emergent design is the design approach adopted by XP, and like other XP practices emergent design relies on the presence of other complimentary practices which are cleverly woven together to reinforcing each other, the whole being greater then the sum of the parts. The practices that come to mind are common code ownership where every developer on the team "owns" all the code and hence is responsible for comprehending all the design. Another is pair programming, where the quality of the code is policed by continuous peer review and design knowledge is communicated across the team by rotating pairs. TDD helps by documenting the design specification as executable tests, the design specification is then kept up to date by refactoring the tests along with the code. Another XP practice is small teams working in a "bull pen" or some other type of congenial space. The small size reduces the number of paths of communication, so if one developer makes a design change he only needs to broadcast that change across a small number of paths to ensure that a fully connected network of design knowledge is maintained. The congenial space facilitates 'osmotic' communication, where the communication of small day-to-day design changes and refactors occurs coincidentally, merely by people over hearing design discussions amongst pairs of programmers, or perhaps by the team breaking out and quickly discussing a design choice facing someone, coming to a joint decision, and going back to work with a new shared understanding.

Its All about People
With small teams seeded with people skilled in these XP practices, the XP approach to maintaining system comprehension amongst a team works very well. It does require a number of subtle skills though, many of which are right brained 'soft skills'. I usually recommend a ratio of 1 to 3, skilled practitioners to novices. With such a ratio I find that the novices pick up the idea pretty quickly. Interestingly Williams discussion on emergent design also identifies the need for teachers and learners, when it comes to mastering the skills needed, and of course we are all learners when it comes to discovering the emerging design of a new system.

Code as a Communication Medium
In my experience once you have experienced this informal means of maintaining system comprehension, you never really see the need for the types of traditional system documentation we have all gotten use to. Novices in these informal communication tools soon become masters themselves and go on to seed other teams. Unfortunately, in many environments we do not have the luxury of creating the right conditions for informal osmotic communication. So what then?

William made the following comment which is thought provoking:

Finally: Documents per se are not evil, but I have come to realize that the documentation tools may not serve the goals. I mean, I need to be able to take a 10000 feet look to a system, to see the big picture. But, I will not be able to see it looking at just code. A word document is of no help. I need something else.

The code is not sufficient? Why? I guess it depends on the prior context of the reader, and also on the quality of the code. In many organisations, people not directly responsible for the code are called upon to make comment and review 'other peoples' code. Now in this scenario the reader has very little context. The XPer would say, well you've broken the tenant of common code ownership. If the reader was part of the team, then he would also be part of the osmotic process and would have a great deal of context to draw upon when reading any given section of code. Yes, but lets assume that the environment is not conducive to "common code ownership". What then?

Readable Code
The other issue is the readability of the code itself. This can be a self fulfilling prophecy: people looking elsewhere other then the code to gain system comprehension, so the code itself need not be comprehensible. For XPers, their code is the main means of design communication, so they are very fussy about their code. For XPers production code and test code together encapsulate a body of knowledge that describes the design choices that they have made along the way. Kent Beck chooses to make a distinction between 'quality' code and 'healthy' code. 'Quality' code is code which has desirable external attributes, like a low number of bugs. Healthy code is code with desirable internal attributes, that allow the code to be maintained, changed and extended even when the team looses people and new people come on board. Healthy code is easy to comprehend and well designed. I don't make this distinction myself. I see health as part of quality, but it is interesting that Kent does make a distinction, perhaps in an attempt to raise the profile of internal code quality which in some environments is readily overlooked.

Domain driven design and the idea of an ubiquitous domain language which is carried through into the code, is yet another method to improve both the comprehension and the design of code. Cryptic names in code have long been seen as a barrier to comprehension, and lots have been written about 'self documenting' code and how to write code which is more comprehensible. So we do know how to write readable code. The real issue is whether readable code is truly valued by the team and seen as a necessity.

My experience is that after being given an overview of a system, perhaps at a white board, if the code is healthy and good package, class, method etc names are chosen and if the code as a good accompanying test suite, then with the help of an IDE, comprehending a system just from the code is possible. For me it normally takes an after noon or two and in the end I usually end up with a list of questions to be answered by someone who "owns" the code. Questions answered, I normally have a reasonable comprehension, and more importantly an understanding of the 'true' design , not what was originally envisioned by the architect at the beginning of the project, but has long since been superseded as the true design emerges during 'implementation'.

Other Documents
Having said this, two days of my time, and perhaps a day of someone else's is a lot to spend on comprehending a system, especially if I do not intend to work on it. If the code is not very readable it could take me a lot longer. It is ironic that the kind of organisation that does not value readable code is also likely to expect people external to the team to comprehend and evaluate the system in an instant purely from documents, forgetting that the code is the system and hence the most important document of them all. In many organisations this scenario exists so what then? I have found the use of a Software Maintenance Handbook very useful in the past. A guide for the explorer who needs to comprehend the system. A user guide, can be most informative too, especially when it comes to answering the question "what problem is this system trying to solve ?" But even with these documents, assuming that they are kept up to date, there will still be gaps in understanding, so what should you do?

Magical Tools
Lets assume that I had a magic wand that would produce the 10,000 foot view of a system at an instant, just by tapping my wand on the hard disk containing the code :) What then? would I really comprehend the system without speaking to anyone else? In my experience the answer is no. I would understand the structure of the system, the 'what', but I wouldn't know the 'why'. To know the why, I need to know abut the problem domain. In a complex domain space, my lack of understanding of the domain, would be a barrier to system comprehension in itself. So even with a magic wand there is no easy answers, and in the end you need to speak to some one, perhaps face-to-face.

Better languages
If your organisation doesn't allow people to take the time to explain the system to you, then perhaps the left side of our brain may be able to help out a bit. Domain driven design and object orientation both promise to help system comprehension through good naming and the use of modularity. Whilst some OO languages are turtles (objects) all the way down, most are not modular when it comes to high level 'architectural' abstractions. Newspeak is a new OO language that has borrowed ideas from Beta to allow you to define abstractions larger then a class. Once you have a language like Beta or Newspeak that allows you to capture high level design decisions in code, then you can use a code browser to project views of just the high level abstractions, filtering out all the details and providing the 10,000 foot view at an instant. Martin Fowler has been talking about such browsers under the label 'language work benches'. Martins focus has been DSLs and their comprehension and use, separate from the host language in which they reside, but the same idea can be applied to architectural layers, modules, components, libraries, packages etc, any abstraction that can be captured by your language and projected independently as a view. The Beta language already shows how this can be done, by allowing a graphical view of system components gleamed from the code. This idea should be familiar to anyone who has used TogetherJ, the Java/UML documentation tool.

Fix the root cause
As you can tell, my view is that system comprehension is often a people problem, born out of the ways people choose to organise themselves and the values and principles they choose to adhere to. The whys and wherefores for any given system should be tacit knowledge within the team that created that system. Tacit knowledge is born out of daily work and socialised informally across the team. It is only when we restrict the flow of tacit knowledge by creating artificial barriers within the team, or geographically splitting the team, or creating teams that are too large, that we create the need for formal communication. Formal intermediate work products are then handed-off from one part of the team to the next. These hand-offs are points of weakness and are best avoided.

Most software organisations are dysfunctional in this regard, by this I mean that their organisation is not well suited to their intended purpose, namely the production of high quality software. The solution? Change the organisation. I accept that most of us aren't empowered to make this type of change, but lets not forget that the root cause of poor comprehension is often self imposed.

Conclusion
So in summary, system comprehension is about communication. Ultimately the code is the design and hence the best medium for communication. With advances in language design the code can be made a better medium for expressing system structure at the 10,000 feet view, but comprehension is not all about structure, it is about meaning and purpose too, and to comprehend these requires a grasp of the domain. Gaining this knowledge often means talking to people.

As human beings competent in applying both sides of our brains, talking and listening as a way of comprehending a system should come naturally. When it doesn't then perhaps its a sign of a deeper problem, such as organisational dysfunction.

9th October 2008 - Expanded the section on 'Fixing the root cause' to clarify what I feel is the main barrier to system comprehension, poor organisation; in response to comments made by William Martinez.

Google Chrome - Vilan or the next logical step?

2008-09-02T18:07:00.000-07:00

Its a strange feeling when an idea you think is gaining momentum suddenly becomes concrete before your very eyes, confirming that you were spot on.

It is not news that Google is a clear advocate of software-as-a-service, but I'm not sure that most pundits thought that Google would be making the bold step of creating their own client platform just yet.

Well Google Chrome is out and I'm sure that there is going to be plenty said about it. Is this the browser wars all over again? Well when I first read the news (on the BBC news website), I thought so, but when I took a closer look at the details released by Google I changed my mind.

As my last post on Flex makes clear, the current browsers as a client platform just aren't up to the job. The JavaScript interpreters are slow and the whole thing is single threaded, so poorly written JavaScript can lock up your whole browser. Flex has addressed some of these problems, but anyone who remembers Windows 3.1, would agree that Flex will never be able to avoid the browser equivalent of a Windows 3.1 blue screen :)

In the noise I'm sure that Googles words are likely to get lost so here is a link where you can read why Google chose to create Chrome and you can decide for yourself whether they are satisfying a real need.

Should we fear Google in the same way we use to fear Microsoft? I don't think so. This snippet from the Google cartoon on Chrome sums up their position for me (click on the image to make it full screen):

So they want to create something better, and we shouldn't be worried about vendor lock-in because "this better thing" will be open sourced. Over the years I have come to the firm opinion that innovation should never be held back by premature standards. Our current crop of browsers where originally designed to solve a different problem, and whilst they are familiar and use a standard metaphor, they cannot be said to address the needs of software as a service.

There is no reason why Googles new platform should not be open, supporting ActionScript and Flex along with ECMAScript (JavaScript). In fact given that Chrome will be open sourced, there is no reason why people can't write a Ruby or Python or even Smalltalk plug-in for Chrome themselves. This is all consistent with the RESTful idea of code-on-demand, which doesn't dictate what is meant by "code". It will be interesting to see just how open Google are with their new platform, and whether they provide extension points so that others can extend the platform to meet their own specific needs.

Infact if Chrome gains momentum, then other browsers like FireFox or Safari could choose to integrate Chrome, or even borrow ideas. We need to see what type of license Google chooses to use, but a BSD style license would allow both open sourced and closed sourced projects to embed Chrome code. Even if the code isn't directly re-used by other browsers then I'm sure the idea of a multi-thread/process, high performance, networked, client software platform will be.

Chrome seems like the next logic step to me, and the fact that it is coming from Google and is open sourced, compared to a closed proprietary solution like Apples Dashboard Widgets, is a positive step forward. We need to be vigilant and hold Google to account, but I think that Chrome should be cautiously welcomed.

Last point, in executing our responsibility to ensure that Google adheres to ethical competitive practices, why not clean up and clarify our language? I am not sure that the vague term "Cloud Computing" is helping, why not use the term "software-as-a-service" (as Google has chosen to do)? Or better still why not stick with the term "code-on-demand" as coined in Roy Fieldings REST dissertation? I believe language is important and labeling these new client platforms in a precise way will provide less wiggle room for marketers looking to exploit vague terms like "cloud computing" to their own ends. Code on demand makes the purpose clear and by using RESTful language, I think it will help to tie these new platforms into open standards like HTTP, ECMAScript and APP (Atom Publishing Protocol). Just a thought :)

Adobe Flex - ActionScript Flexes its Muscles

2008-08-20T13:51:00.000-07:00

Recently I've developed a keen interest in software-as-a-service and what as been coined Cloud Computing. The idea is to use current web technologies and RESTful principles to provide software services to a larger audience. This has led me to take a more detailed look at Flex and its associated technologies. You can only get the best out of Flex by using ActionScript in conjunction. ActionScript is a dialect of JavaScript or to give it its proper name ECMAScript. JavaScript is a late-bound, dynamic, prototype based OO language. Its worth stating this because although most people are familiar with JavaScript, very few know its OO origins.

JavaScript is heavily influenced by Self and Scheme, a point brought home by its original name "LiveScript". The name JavaScript was dreamt up later by Sun and Netscape during the Browser Wars. Given its roots, I have always thought that JavaScript is under utilised and unfairly maligned. Beyond DOM hacking on the browser very few people know anything about JavaScripts OO credentials which are pretty impressive. Most peoples experiences are coloured by cross Browser incompatibilities and poor performance. Recently Dan Ingalls, of Smalltalk-80 fame, has taken JavaScript and shown what it can really do. This work has resulted in the Lively Kernal. Its worth taking a look at what Dan and his team have achieved. It is pretty impressive.

Over the years ActionScript as migrated away from JavaScript in an attempt to become "Java Programmer" friendly. Well its succeeded. ActionScript code looks very similar to Java. You can use type annotations and classes just like in Java. It also supports packages and limits you to one class per file, so it is really home from home for the average Java programmer. The idea is to provide an easy on ramp for developers moving to ActionScript from C++, Java and C# .

Having played with Self and seen what it can do, I was curious to know how much of Selfs dynamism had made its way into JavaScript and ActionScript. Well firstly, JavaScript retains slots just like Self. So JavaScript has representation independence. Unlike Self, a JavaScript Object only has a single parent slot, so no multiple inheritance. Oddly enough the parent slot is called the 'prototype' which I find rather confusing. In Self the parent slot is called the 'trait'.

Objects are created in JavaScript using constructor functions. Functions in JavaScript are first class objects, again something borrowed from Smalltalk/Self or is it Scheme? I'm not sure, but functions are objects also. If you must use a C-like syntax, then JavaScript is a pretty impressive son of Self in my opinion, inheriting most of the fundamental qualities of its father.

OK. How well does ActionScript stack up? ActionScript is a difficult language to get into. Macromedia/Adobe assume that all you want to do with ActionScript is web presentation. This means that there doesn't seem to a standalone interpreter you can use to play with ActionScript just on its own. Well I've succumbed and installed Flex Builder, the Flex IDE, and have been rather impressed with it. The "main" for ActionScript applications is the "onCreationComplete()" method hook inside a Flex web page. Without going into the details, you need to write a Flex app to use ActionScript. If anyone out there knows how to run ActionScript standalone then I would be very grateful to find out.

Fortunately FlexUnit does all this for you and you can get playing with ActionScript very quickly by writing tests. OK, back to the ActionScript language. I started writing JavaScript and compiled it using the ActionScript compiler in Flex Builder and it works. So ActionScript does everything JavaScript does.

What they have done is extended JavaScript without fundamentally changing the internals. I concluded in my blog post on classes versus prototypes, that prototypes where the more fundamnetal abstraction. Well, ActionScript proves this. They have added class declaration syntax mostly as syntactic sugar for the creation of prototypes. They have taken this further in ActionScript3.0 (AS3), changing the method lookup mechanism, by copying down method objects from the class hierarchy into a single 'trait' object at compile time. This speeds up method dispatch. I don't understand all the details, but the only restriction introduced is that an objects class is immutable. Classes themselves are still open, just as in Smalltalk or Ruby, so AS3 is as dynamic as these two class based languages (with the exception of 'become:' in Smalltalk). It could be argued that AS3 is not as dynamic as JavaScript. The thing to remember though is that class immutability only applies if you use the 'class' declaration. If you stick to prototypes then ActionScript is as dynamic as JavaScript and Self.

I am pretty impressed with ActionScript. Adobe/Macromedia have managed to extend a prototype based language, making it conceptually similar to C++, Java and C#, whilst retaining the power of prototypes under the covers. It is some achievement.

Even the type annotations in ActionScript are optional, just as advocated by Gilad Bracha. With or without type annotations Flex Builder does a very good job at auto-completion, using type inference no doubt. When you leave out the type annotations Flex builder provides warnings, but these can be ignored or even suppressed. So you can save key strokes if you wish whilst prototyping.

The main problem JavaScript has faced in the past is slow interpreters and incompatibilities across browsers. With ActionScript 3.0 Adobe has solved both of these problems. The Flash VM is ubiquitous with 98% market penetration. The copy-down method lookup mechanism in AS3 provides vtable like performance, whilst still retaining dynamic dispatch when needed. The history of the evolution of OO support in ActionScript and an explanation of the method dispatch mechanism in AS3 is provided here.

Flex/As3 is definitely a web technology to look out for. Its nice to see the work of the original Self and Smalltalk researchers still living on, and playing its rightful place in the future of the web. Good ideas don't die, they just mutate :)

Dynamic Languages - The FUD Continues...

2008-07-13T03:40:00.000-07:00

Cedric Beust has been spreading his anti-dynamic language propaganda in a presentation that he made at JAZOON. I really find it strange that someone who is respected in the Java community would spend so much effort trying to discredit alternative languages. In an attempt to set the record straight for the "busy" Java developer I tried to enter the comment below. Unfortunately Cedrics blog wouldn't accept what it felt to be "questionable content" so I have posted my comment here instead:

Hi Cedric,

I didn't mention why I believe there is so much Fear Uncertainty and Doubt when it comes to dynamic languages. Two reasons: Politics and Fear.

Dynamic languages have been hugely successful dating back to the 1950's so whether they are viable or not should be beyond debate.

So why are we debating?

The real issue is the right tool for the right job. The problem is that lots of programmers only know how to use one type of tool so aren't in a position to make an informed choice.

This inability to choose based on ignorance leads to fear. The politics comes from proprietary languages (Java, C#) where the proponents have a vested self interest in keeping us all fearful of alternatives.

I have been using dynamic languages (Smalltalk) since 1993 and Java since 1996, and I started out programming with Modula-2 and C in 1987. They all have their strengths and weaknesses and none of them are a silver bullet.

The simple truth is that for web applications dynamic approaches are massively more productive. Take a look at Seaside (Smalltalk), Grails (groovy) or Rails (Ruby) and its clear that Java has nothing to compare. The DSLs provided by these languages make web development a cinch. Productivity improvements of 2-3 times is not uncommon. This translates to a reduced time to market, and better response to business needs.

So the real question is why are these languages excelling in this way? You seem never to address this issue, assuming that the people that choose to use these languages are some how misguided or confused. Well they've been misguided since 1956 and the advent of Lisp :) They choose dynamic languages because they value an higher level of expression, allowing them to achieve more with less. This doesn't only apply to the web, it applies to any scenario where a high level, domain specific language is applicable.

You advertise your talk as a guide for the busy Java developer, yet you do very little to educate him and alleviate him of his fears.

Let me:

1. Programming is hard and there are no Silver Bullets.

2. The biggest determining factor for success are the skills of the programmers.

3. Dynamic languages are different, requiring different skills and a different programming style.

4. If you take the time to master these skills then you are in a position to choose the right tool for the job: Either static or dynamic, or perhaps both.

5. With the right skills and the right tools you have a built in competitive advantage. Human centric computing applications call for higher level languages. Dynamic languages allow for the creation of higher level domain specific languages in a way that static languages don't.

The last point deserves to be backed up. Take a look at the Groovy HTML builder as an example and compare it with Java JSP. An even better (all though more esoteric) example is Seaside in Smalltalk.

The domains where Java makes sense are shrinking. Given the performance of dynamic languages nowadays and the ability to inter-operate with lower level, high performance system languages like C/C++, I see Java and C# being squeezed.

If you want productivity and a higher level domain specific language then Ruby, Groovy, Python etc is a natural choice. If you are on the JVM or CLR then you can always fall back to Java or C# when you have to. If you are on a native VM then you can fall back to C/C++.

The right tool for the job, which may mean splitting the job in two (front-end, back-end) and using different tools for different parts. With messaging systems and SOA "splitting-the-job" is easy.

Dynamic languages will only get better, incorporating better Foreign Function Interfaces and better tooling support, in the same way Java did back in the late 90's. BTW adding type annotations is always an option if people think they are really needed, but like I say a sizeable community have thrived very well without them since the 1950's :)

Cedric. You do your self no service by dressing up your prejudices as scientific fact. How about a balanced expose?

Go on surprise me :)

Paul.

The Self Language

2008-07-11T00:10:00.000-07:00

In my last post I said that I wasn't too impressed by Selfs idea of using prototypes. Well I've changed my mind. My initial complaint was a lack of structure. When you open the Self BareBones snapshot all you get is a graphical representation of a shell object and a waste bin object. You don't get to see all the classes in the image like you do with the Smalltalk Class Browser. There is no Class browser in Self because there aren't any Classes.

This doesn't mean there isn't structure though. If you get the lobby object you will notice a slot called globals, one called traits and one called mixins. As I mentioned in my last post traits are objects that are used to encapsulate shared behaviour (methods). Globals are a parent slot on lobby; inside globals are all the prototypical objects. Each prototype object has a trait object. The prototype holds state whilst the trait holds behaviour. So between the two you have the same structure as a Class. So you create new objects by copying prototypes which inherit shared behaviour through an associated trait object.

Since the traits slot is not a parent slot of lobby you must send the message 'traits' to access trait objects from the lobby. So 'traits list' gets you a reference to the list trait object and 'list' gets you the list prototype. Why is the lobby important? Well all new objects are created in the context of the lobby. So the lobby object acts like a global namespace.

My explanation makes it sound more awkward then it actually is in practice. The bottom line is that Self has a lot of structure, as much as Smalltalk in fact. The structure is just different and more granular. Working with this structure is actually very pleasant. You still think in terms of classes, but only after thinking about the object first. So with Self you create a prototypical instance of what you want then you refactor it into common shared parts (traits) and an instance part (prototype).

The traits are more or less Classes. Self objects support multiple parent slots, but by convention multiple inheritance is not used. Instead usually there is one parent trait and additional shared behaviour is achieved by adding mixins to additional parent slots.

I am beginning to agree with Dave Ungar, that the Self way of thinking about objects is more natural and more simple. What convinced me is the ease in which objects can be created:

( | x <- 100. y <- 200 |)

Is an object literal which you can create at the shell and get an instant graphical representation of. The graphical representation of an object in Self is called an Outliner which is basically an editor that allows you to view, add and modify slots on the associated object. The Outliner also has an evaluator, where you can type in messages and have them sent to the target object.

So in self you create objects by entering literal text, test them out by sending messages and extend them by adding new slots. This is all achieved with instant feedback. You then factor your objects into traits and prototypes by creating new objects and moving slots through drag-and-drop.

Is all this important? I'm not sure, but I think so. The fact that objects have a literal representation that includes their behaviour is quite interesting and I like the drag and drop refactoring. What I can say is that the Self approach is fun and feels more concrete, as if you are using real building blocks to create your program.

Would a Self approach lead to higher productivity? With a bunch of keyboard accelerators so that you didn't need to use the mouse much, then I think so. To me feedback is king, and Self offers plenty of feedback. I think that the Self approach also leads to a more exploratory programming style, which in my opinion is a good thing. Above all, manipulating objects as if they are 'real' is a lot of fun, which as got be be worth something in itself :)

Objects revisted - Don't Generalise?

2008-07-04T01:48:00.000-07:00

I've been playing with Self and it has got me thinking about why prototype based OO is not more prevalent. Generalising and categorising a bunch of things, as all being the "same thing" is something we all do all the time. Yet we know that we shouldn't generalise this way since each individual "thing" is unique :) I have come across this paper that takes a philosophical look at the difference between prototypes and classes. It concludes that ultimately prototypes are more expressive but generalising into classes is "good enough" most of the times.

Just from a practical view point I find classes much easier to work with thus far. This could be due to my vast experience with classes versus prototypes. Classes impose structure which aids with comprehension I find. I need to play with Self some more, but at the moment I find myself translating Selfs idea of a parent "trait object" into the more familiar concept of a Class. Here is another paper that takes the opposite point of view, stating that prototypes are more useful on practical grounds.

The motivation for prototypes as I understand it was the fragile base class problem. Representational independence and mixins largely solve this problem. Bob Martin takes another slant on the idea of a brittle base class, by stating that base classes should be stable by design. So the fact that other classes depend heavily upon them should not cause a problem. Classes that change should not be base classes. So base classes should encapsulate stable policies.

One thing that is clear to me is that classification and classes can be viewed as an additional structure imposed upon an object (prototype) based environment. So prototypes are the more general mechanism. The Self image I have been playing with has an emulated Smalltalk environment built from prototypes. So based on this, prototypes are the more fundamental abstraction. Following this logic, then Lisp with it's multi-methods and macros (code as data) is also more fundamental and hence more expressive then a prototype based language.

So it all boils down to Lisp :) I guess what we have with OO is a language imposed structure that enforces the encapsulation of state. This structure reduces the gap between the base language (Lisp like) and the problem domain in many instances. So OO itself can be considered a domain specific language, were the domain is "the physical world". In many scenarios, classifying objects and sharing a common behaviour (class) object across a set of instance objects is an approach that maps well to how we mostly think about the world and hence is a "good enough" template with which to model our world in a number of useful ways. But we know from philosophy that classifications aren't concrete and are arbitrary to some degree. If we choose to apply this deeper appreciation of the world around us to our models then we must do away with some structure. To model a world where objects aren't constrained by classification, we can choose to use prototypical objects, allowing for object specific behaviour.

So there appears to be a trade off between structure and expressiveness. So it follows that we gain more flexibility and freedom of expression if we fall back to a language with a less rigid structure, like Lisp were we are free to model the problem in any way we wish. The downside is that we now have to take more responsibility for structuring our program ourselves.

The bottom line is usefulness I think. For most use cases prototypes do not seem to provide any additional utility over classes. I'm curious to know whether there are uses where prototypes excel. From what I've seen so far, the Self Demo could have as easily have been written in Smalltalk.

(An after thought: Ruby also allows you to add behaviour to specific objects. This is not the same as Self, since a Ruby Object must still have a class and doesn't have parent slots where it can inherit traits dynamically).

Newspeak - The Hopscotch IDE

2008-06-12T04:41:00.000-07:00

In my last post on Newspeak, I ended with the conclusion that I needed to further my education. Well, I've broken the back of Haskell and pure functional programming so I feel confident commenting on Newspeak again.

First thing to say is that I feel silly for doubting Gilad and his team in the first place. There is a lot I could say about Newspeak, such as the benefits of using Newspeak in a pure functional style with the associated ability to exploit multi-core processors, but I won't. There is now a paper on Newspeak and I am sure that many others will pick up on these salient points. No the thing that has blown me away is this paper by Vassili Bykov on the Newspeak IDE interface called Hopscotch.

I'm still half way through the paper, but Vissali's style and approach reminds me of when I first read the blue book by Adele Goldberg. In a world were we have all become pinned in by our prior context, Vissali as the ability to see the user interface problem afresh and has come up with an elegant approach which emphasises usability and productivity.

His argument is that the "form" metaphor that we have all become accustomed to, is inherently modal and leads to a fragmented user interface. Instead Vissali chooses to borrow the document metaphor from the web, which he extends so that it is domain object aware. So in Hopscotch a class becomes a document containing other domain objects such as nested classes, methods etc, each of which have their own presentation. The IDE is aware of the structure of the underlying class, and orchestrates show/hide, and navigation between the sub components of each class. Vissali explains this a lot better in his paper.

My main reason for blogging is that I haven't read something that sounds so appealing, elegant and obviously right, since when I first read the Blue Book on the Smalltalk language. To be honest, I haven’t felt so excited about software in a very long time indeed. I can’t wait to get my hands on Hopscotch, which hopefully should not be too long now, since Newspeak will be open sourced. I have a very strong feeling that Newspeak is going to turn out to be something very special indeed.

Haskell - Academia goes Bowling

2008-05-09T23:14:00.000-07:00

True to my promise, I've spent a little time looking into Haskell. I've read some intros on the web, and I have even gone as far as to download HUGS an interactive interpreter, and I have started to work through a tutorial. I'm still on the first chapter so it's still early days.

First impressions are that I like it. Haskell as you would expect is very mathematical, with terms like "currying", "list comprehensions" and "guards" to learn, so I had half made my mind up not to like it. Then I started to use HUGS and actually started writing some Haskell code. The code itself is quite readable, helped by prefix notation and type inferencing I think.

Keen to get to the point (and not wanting to have to learn a bunch of mathematical theories to do so), I sought out a real world example of a Haskell program. Thankfully I didn't have to look far. Ron Jefferies has produced a number of very good articles where he walks through his experiences coding up a Bowling Game program using TDD and a number of different languages. The Haskell example (produced by an academic), drew a lot of interest, especially since Ron spoke disparagingly about it.

The program uses pattern matching and recursion, avoiding loops. Ron's point was that the recursive implementation although succinct, said nothing about the "tenness" of a game of bowling. For those who don't know a game of Bowling always consists of ten frames. This "tenness" of the problem wasn't expressed anywhere in the solution. A number of academics jumped in defending recursion and offering recursive solutions of their own. Until one guy found out that there was actually a bug in most of the implementations presented.

I'll leave the reader to follow the debate, but what all these academics had missed, and Ron with his practical instincts had sensed is that the programmer is always the weak link. Any solution that doesn't fit with the way the programmer thinks about the problem will prove difficult for the programmer to verify in his own mind. Hence the missed bug, despite, many "clever eyes" over the code. Ron's summary is that Haskell forces you to think the way it thinks (mathematically) rather than allowing you to express things the way you think.

Gilad makes the same point about pure functional languages. In contrast, pure object orientated languages allow you to model the world the way it is. Haskell encourages you to create a stateless functional model, which by itself is useless for real world problems, but supposedly great for verification tools. Where your pure functional solution meets the real world, for things like user I/O, you must fall back to impure actions. Impure imperative behaviour can be isolated from the rest of your code using Monads. I don't yet fully understand Monads, but they appear to be a way of representing an underlying type (e.g an Integer) by wrapping it in a new type (called a Monad) that provides additional values that transmit information about side effects. So for example "Nothing" (NaN) is a valid value for a Maybe Monad, which can be used as a wrapper type for Integers. So functions that return Monads are able to communicate side effects to other calling functions (meaning that the side effect is no longer a "side effect" but a legitimate return value) . I will blog more about Monads once I fully understand them.

So where are all these great verification tools? With the exception of a type system there doesn't seem to be any built into the language itself. So you end up with a program that is hard for the programmer to check himself, and which can't be fully checked by the compiler. So what's the point?

As an academic exercise, Haskell is interesting and I guess it will float the boat of the mathematically inclined, as a practical tool though, it wouldn't be at the top of my list. I'm still going to play with it some more though, and I'll let you know if I change my mind.

Updated 11th May 2008
Corrected my description of Monads after feedback from the Haskell community. I found Daniels comment on Monads particularly useful. It prompted me to read further and improve my own understanding. I don't pretend that my description of Monads is complete, and I would refer a reader interested in a complete explanation to look here.

Newspeak - Educating the Pilots

2008-05-05T03:43:00.000-07:00

In my Man versus Machine post I decided to play devils advocate and question Gilads motives. Having seen Gilads presentation on Newspeak, his motives are now clear to me, and as far as I understand his intent, I am in violent agreement. So why my initial discomfort?

Before I answer this I should say, that I have always agreed 100% with Gilads' point on static. On his blog I questioned whether the elimination of static, was just another way of reducing coupling, to which Gilad responded that yes this is true, but that coupling was a general term and that he preferred to be specific. I think this is the root of my discomfort. By being specific, I fear that a lot of normal programmers will miss the point.

Most programmers do not understand the importance of coupling and cohesion, and most do not understand that objects are just one approach to reducing coupling and increasing cohesion amongst abstractions. Functional programming is yet another approach. The underlying problem is the same, and as old as computer science itself. In fact the whole static thing is merely a failure in abstraction. We have no way of declaring a suitable cohesive abstraction, so we make things static and have them coupled to everything. Just like dodgy globals in C when you can't think of a better way.

So what Gilad has done in Newspeak is to provide a new way of defining loosely coupled, cohesive abstractions that are larger than a single class, removing the need for dodgy globals (static). This isn't reducing the programmers freedom, rather it is replacing a rather poor default mechanism, with a more precise way of defining exactly what you want. If you wanted to you could still create "dodgy globals" in Newspeak, but you would need to define a "global" abstraction first. There is no default "global" name space in Newspeak, which is as it should be.

So this is the issue, new and improved mechanisms are fine, as long as the pilots understand why? With OO programming, lots of programmers understand the "what", but very few understand the "why". This is how languages like Java can claim to be Object Orientated whilst only delivering on half the story. It is also why many (most?) Java programmers are happy programming in a procedural style in blissful ignorance.

With Smalltalk, it gave very little scope for misunderstanding, because there was only one way to do things. You were constrained to think in Objects. Newspeak has this same property, and by replacing static with a better mechanism it will force developers to think about higher level abstractions and name spaces, rather than defaulting to static.

I guess it isn't Gilads job, but there is a big education task needed if languages like Newspeak are to ever take off. Most people don't get the basic concepts behind OO and Smalltalk, whilst Gilad is ramping things up to the next level by borrowing from Self and Beta. My education is only partial. I have read about slots in Self and they make sense. I didn't think of them as a way of defining immutable functions though, and as a way of eliminating state. The elimination of state, similar to the elimination of static is yet another way of reducing coupling. If nothing changes, then any dependent abstraction can't be adversely effected by unsuspected side effects. This intuitively makes sense.

So Gilad prefers:

identifier = letter, (letter | digit) star. (Immutable)

Over:

identifier ::= letter, (letter | digit) star. (Mutable)

In both examples identifier is a slot. This means that identifier is a method, that answers the value of the given expression. There are no instance variables in Newspeak. In the first example identifier is immutable, in the second it is mutable. In this presentation at Lang.NET Gilad makes the point that if you remove the ::= operator from Newspeak, then Newspeak becomes a purely functional language with no state.

What are the implications of this? Well this is where I hit the bounds of my education. I think I'm going to need to take a look at a purely functional language like Haskell and get to grips with Nomads before I can comment further. Even with a vehicle as powerful as Newspeak, the limiting factor on success is still the pilot :)

To the Victor the Spoils

2008-05-03T14:21:00.000-07:00

Yardena kindly supplied this link to an interview with Erik Meijer, Gilad Bracha and Mads Torgersen, talking about Programming language Design and Evolution. The interview was interesting with Gilad confirming something that I have suspected for a while, which is that state is optional, and that most/all? problems can be decomposed into immutable functions. The problem with pure functional programming though is that it is really hard to do well, simply because we just don't think of the world this way. We think of the world as having state and we find it easy to model the world as stateful. Gilad uses the example of the interviewer as being a thing that is "stateful", rather than being a sequence of immutable things each one with no prior memory, only existing for a given moment in time.

Given that programming is hard enough as it is, relaxing the constraint of "no state" and introducing the idea of encapsulated state is a big leap forward I think, and represents a great, still untapped opportunity when compared to "anything goes" procedural programming. Contrary to the "OO" marketing hype over the last decade, the dominant programing paradigm today is still procedural in my view. The direct consequence of procedural based hybrid languages like C++, Java and C#, that only pay lip service to OO and functional ideas, whilst lacking the uniformity to be truly functional or Object Orientated.

It was interesting to hear John McCarthy say that the idea of encapsulated state, as introduced with Smalltalk has been the only significant step forward in programming language design in the last 50 years since Lisp. It was painful to watch the interview as the interviewer made claims for languages like C++. Even going as far as claiming that Mac OS X was written in C++ (although OS X is written in C and Objective-C), and that we still need C++ for all our "Systems" programming needs today. Gilad winced more than once. Which prompted the question: "Why is it that "beautiful" languages aren't more popular if they are so much better?". Gilad tried to answer this one, using a form of Alan Kays "prior context" argument, saying that programmers didn't like new ideas, and that change needed to be introduced incrementally.

Interestingly it was the C# guy Mads, who debunked this half truth, by saying that programmers were interested in new ideas, but that "new thinking" didn't pay the bills. Mads also admitted that a lot of the work that lead to the dominant static VM platforms of today was based on wrong thinking stemming from compiler based, code optimisation ideas, and it would have been much better to have started off with a dynamic platform and to have layered on top an optional type system, much like Gilad advocates. He puts this realisation down to hindsight, forgetting to mention that people like Alan Kay and Dan Ingalls were advocating just this at the time.

The biggest disagreement was over the future role of C++ with Gilad claiming that the language was as good as dead, and that they should have stopped extending it long ago. The others, including Erik, felt that it was still useful with advances in static analysis tools still being made today. It was painful for me watching advances in IDEs and Operating Systems being attributed to C++ and it's prodigies Java and C#, when it is clear to anyone with any grasp of history that we owe the modern Graphical, Windowing Personal Computer world to Smalltalk. You could see the pain on Gilads face as he sat through this mis-representation of history, mostly by the interviewer.

Mads made the point, that the successful languages today are all incremental changes to the thing that existed before them, namely C. And I guess this is the rub. C/C++, Java and C# are all successful. Smalltalk is not. To the victor the spoils, which means that the victor gets to write history.

This is fine with me as long as Smalltalk gets to write the future :)

Newspeak - First Impressions - Underlying Beliefs

2008-04-26T00:47:00.000-07:00

First thing to say, after watching Gilads presentation on Newspeak, is that I feel that I'm not qualified to deliver a rigorous critique. This in itself raises an interesting question. After 18 years software engineering experience my "education" is insufficient to evaluate a new OO language that builds on ideas that are over 20 years old. Why? Well, my professional career (the usual diet of Java APIs and frameworks) hasn't prepared me for Newspeak. In fact it is only my self education which I performed at home in my spare time that allows me to grasp most of what Gilad is speaking about. This telling fact says more about our industry I believe then it does Newspeak.

Beliefs and Motivation
You may find beliefs a strange place to start an evaluation of a programming language, but I think that the beliefs of the protagonist behind a language lay at the core of what a language is about, and whether it will appeal to me. Language design, like any other form of design is about trade offs, and it is your beliefs that determine the relative value you place on competing ideas and principles. Well the first thing to say is that Gilad is still a Smalltalker at heart, and he describes Newspeak as a direct decedent of Smalltalk. The other significant influences he mentions are Self, Beta and Scala in that order of significance. Scala seems like the odd man out here, but when you realise that its influence is much smaller then the others (the formalisation of object constructors, which is something Smalltalk never did), then the idea of Newspeak as a successor to Smalltalk makes sense.

My concern was whether Gilad believed in pandering to the needs of the tool over the beauty of the language from the perspective of the programmer. Well he makes his feeling clear in is preference for:

id = letter, (letter | digit) star. (Newspeak)

in comparison to:

id = letter().seq(letter().or(digit()).star(); (Java)

When attempting to express the BNF Grammar:

id = letter, (letter | digit) *

He refers to the redundant parenthesis in the Java version as "solution space noise". He says that what we should be doing is expressing our problem. Stuff in the code that is there to satisfy the tool (Compiler) is noise and should be reduced to a minimum. This belief sits fine with me. It also fits with my artist analogy. An artist is more interested in his subject then he is in his tools. Languages like C++ and Java just miss this point, and when I think about it, this is the central idea I learned from my first encounter with Smalltalk all those years ago. Just going through Squeak the other day I stumbled on things like this:

(2 + 3i) sqrt.

What do you think this does? Yes, you guessed it. It answers the square root of the complex number 2+3i. Now imagine expressing this in Java?

OK, if Smalltalk is so beautiful what is left to be solved in Newspeak? Well plenty actually. In terms of beauty, Smalltalk has its warts too. Like having to place the word "self" before each message send when sending a message to your self. Newspeak, like Self makes the receiver as "self" implicit.

The previous BNF grammar expressed in Smalltalk is:

id := self letter, ((self letter ) | (self digit)) star.

Which is more pretty then Java but not as pretty as Newspeak.

Smalltalk also lacked the ability to define encapsulated abstractions that were larger than a single Class. So although Smalltalk is uniform and "turtles" all the way down, when it comes to Classes everything is horribly coupled in a single global hashmap. Newspeak solves this lack of modularity problem too. So Newspeak is turtles all the way down and all the way up as well, leading to pluggable libraries etc. It achieves this by borrowing the idea of nested classes from Beta. So a module or a library is a class that contains other nested classes. I'll discuss this feature in more detail in a future post, but it is very significant and Gilad highlights it as the major difference between Newspeak and Smalltalk. It allows you to swap out abstractions at a category or library level with Newspeak without worrying about ugly inter-class dependencies (coupling). So Newspeak can be shrunk as well as extended unlike Smalltalk.

I've said that Smalltalk is turtles all the way down. Well that isn't quite true. All OO abstractions (turtles) should encapsulate their insides from the outside world. Smalltalk does do this for member variables, but in Smalltalk-80 all methods are accessible from outside the class in which they are declared, including methods that are meant to be private. When I first came across this, I thought it was a minor oversight that would be fixed soon, but interestingly the situation is still the same today in Squeak. Gilad intends to fix this oversight in Newspeak. So one of Gilads motives seems to be to finally address the design flaws in Smalltalk.

I find that many of Gilad's throw away ad-lib comments are more informative then much of his precise technical explanations. Throughout the presentation, and especially when he would switch to his Newspeak IDE, Gilad would make reference to the past failures of Smalltalk, and how he hoped to address those issues in Newspeak. In the eyes of many, Smalltalk never really made the transition from a research project to an industry ready software engineering platform. Strongtalk was one attempt to complete that transition. Strongtalk dropped the "many windows" class browser approach of Smalltalk-80 and adopted a collapsible pane approach that would be familiar to users of Eclipse today, and feels more natural to people coming from a "file" based metaphor for organising classes. Newpeak does the same. Thankfully Newspeak drops the type annotations of Strongtalk, with Gilad admitting that many of the late bound scenarios that Newspeak is designed to address would be almost impossible to verify statically. Gilad also plans to have a much improved FFI (foriegn feature interface?) for Newspeak, when compared with a typical Smalltalk dialect like Squeak.

Smalltalk as always been an island. Once you were on it the weather and the scenery was great, but you couldn't bring anything with you from the outside world except your bathing trunks, and you found yourself missing a bunch of stuff you had gotten attached to. In contrast Ruby fits into the Unix ecosystem like a dream, and you can use almost any windowing system, foreign language library and API available under Unix(/Windows/Mac OS X) from within Ruby. You can also "shell out" to a Ruby script from a Unix program too, so their is two way communication. Gilad spoke about a similar ambition for Newspeak with a FFI that allowed callbacks from foriegn functions, although he didn't go into this in detail. He does mention the use of Actors as a means of inter-process communication, so perhaps this is a pointer to how some integration may be done.

So to sum up, Gilad believes in clearly expressing the problem over assisting the tool (Man over Machine) and uniformity and modularity ("turtles") all the way up as well as all the way down. He is motivated by creating a successor to Smalltalk-80 that complies with sound software engineering principles and interfaces well with the current incumbent technology ecosystem. His aim is to create a language which is both esthetically pleasing and software engineering sound. The end goal is an elegant and consistent software engineering platform for use in industry.

I was going to give my impressions on how well Newspeak in its current incarnation actually delivers on these lofty goals, but this post is already long enough. Besides you can watch the presentation for yourself :) I'll be coming back to Newspeak in future posts, where I intend to pick up where I've left off here.

Update 29th April 2008
Gilad has kindly offered the following corrections and clarifications:

Re: your review. Well thanks! A quibble or two:

1. The basic FFI with callbacks is working; the higher level convenience layer isn't there yet. We are just calling to/from C (or Objective C) at the moment. Using actors to interface to the outside world was Alan Kay's original plan, but it was far ahead of its time.

2. Types. I haven't given up yet. It's just our lowest priority and a very hard problem. I would like to support pluggable types in Newspeak, and we currently support and encourage type annotations even though they are not checked.

Making Great Music

2008-04-25T03:53:00.000-07:00

Thinking out load on the web is a risky business, but it has it's benefits. With a little help from your friends it can lead to a deeper insight. The logic of the case made by Yardena and Andrew in response to my original post on Man versus Machine made sense, but it still left me feeling a bit uncomfortable.

Had my artistic analogy run out of steam?:

I love reggae music. I grew up on it: Bob Marley, Dennis Brown, Gregory Isaac, the reggae greats of the 1970's. In recent times Reggae as become more commercial with modern"artists" placing less emphasis on spirituality. I find this change rather regrettable, because without spirit the music becomes just noise.

Then I got to thinking more about those old Reggae classics. If you listen to the original vinyls the music is punctuated by pops and clicks, and even drop outs in places. Sir Clement "Coxsone"Dodd, the owner of the Studio One record label had humble facilities at his recording studio in Kingston Jamaica. I believe many of those originals were recorded using nothing but a simple eight track in a small unsound proofed room.

Some of those classics have been remastered, which gets rid of the pops and clicks, but sometimes you loose a bit of the feel too. Don't get me wrong those old classics are still great pieces of Music, but imagine how they would have sounded if Clement Dodd had the benefit of Modern Recording facilities back then? Great Musicians and Singers with great tools. Hmm...

The Answer: Man and Machine in Unison

2008-04-25T00:28:00.000-07:00

In my last post I proposed the question. What is more important the Tool or the Man when it comes to software development? It was in response to Gilads attempts to eliminate certain dangerous "features" from programming languages. I questioned whether this was the right focus and I got some interesting responses.

This one in particular rang a bell:

Rather than looking at it as Man against the Machine, I see this as a challenge of building a bridge between the two, giving both sides respect and attention. I agree with Andrew - if the machine can't solve the problem it should delegate to the person, but do it simply and clearly. In a more philosophical sense, I think because computer science is so young, we have not figured out properly how to transition from "science" to "engineering" yet. To do this we need tools that are controllable and safe alongside creating good programming curricula to train "the pilots".

I think both Andrew and Yardena are right in their responses. It isn't necessarily one or the other, Man versus Machine. We need both working in unison.

Who rules - Man or Machine?

2008-04-23T09:38:00.000-07:00

I've been following Gilad Bracha's blog on programming language design. The concepts Gilad touches on are profound and his blog is a real interesting read, I recommend it.

Some of Gilads most recent posts have left me pondering whether he is focusing on the right things? Gilad calls himself a Computational Theologist. This is an interesting title since theology has to do with beliefs and belief systems. From Gilads posts on Monkey Patching and Cutting out Static, I get the feeling that Gilad believes that the machine should 'help' the programmer to do the right thing. What beliefs that underpins such an assertion? Perhaps such a believe stems from an acceptance that most programmers are poor to mediocre and need all the help they can get? Maybe this believe stems from an idea that a computer program is more a piece of mathematical logic then a piece of creative art?

I am assigning a lot of beliefs here to Gilad, which are probably things he doesn't believe. The point remains though, at a fundamental level we have the choice of either believing in people or believing in machines. Andrew has a post on Smalltalk where he refers to the original Smalltalk Byte article. The Smalltalk researchers had a human powered approach to 'computer research'. Reading their paper it is clear that they believe in people and our inherent creative abilities. To them the machine is merely a tool. Man, the creative artist exploits tools and mediums to express himself. A computer is merely one such tool with a unique set of characteristics. The purpose of a programming language is to allow Man to exploit the Computer.

It was interesting to hear John McCarthy say a similar thing. For example he rubbishes the commonly held belief that "goto" is evil and should be banned. As an artist I'm not sure how I feel about people banning things and limiting my expression. Imagine a word processor that didn't allow you to use certain words like "fuck" or "piss"? There is a strong argument that such words aren't the most effective form of communication , but can we say they should be banned in all instances?

There is an interesting point of debate here, and I have no firm conclusions, other than to say that people have far more potential then machines. At best machines can help people to explore their full potential and extend their influence and reach. But a machine has no consciousness, no intelligence, no imagination. A machine has no spirit.

As humans we have two parts to our brain. A logical side, and an emotive artistic side. I believe that both sides are involved in everything we do, including programming, and we need to appeal to both. This means that the most correct and safe programming language may not necessarily be the most useful.

I love reggae music. I grew up on it: Bob Marley, Dennis Brown, Gregory Isaac, the reggae greats of the 1970's. In recent times Reggae as become more commercial with modern"artists" placing less emphasis on spirituality. I find this change rather regrettable, because without spirit the music becomes just noise.

What Really Matters?

2008-04-17T01:22:00.000-07:00

My posts of late have been pretty opinionated and uncompromising. Why? Well as a Software Industry we are pretty good at creating reasons to do a bunch of stuff that doesn't really matter. I guess doing this other stuff is easier then tackling the difficult task of doing what does matter.

Anyway, I stumbled on this Quote on the C2 Wki:

What really matters?

Software is too damned hard to spend time on things that don't matter. So, starting over from scratch, what are we absolutely certain matters?

1. Coding. At the end of the day, if the program doesn't run and make money for the client, you haven't done anything.
2. Testing. You have to know when you're done. The tests tell you this. If you're smart, you'll write them first so you'll know the instant you're done. Otherwise, you're stuck thinking you maybe might be done, but knowing you're probably not, but you're not sure how close you are.
3. Listening. You have to learn what the problem is in the first place, then you have to learn what numbers to put in the tests. You probably won't know this yourself, so you have to get good at listening to clients - users, managers, and business people.
4. Designing. You have to take what your program tells you about how it wants to be structured and feed it back into the program. Otherwise, you'll sink under the weight of your own guesses.

Listening, Testing, Coding, Designing. That's all there is to software. Anyone who tells you different is selling something.

-- KentBeck, author of ExtremeProgrammingExplained

If I had my way these four points would be prominently displayed in every IT Department out there.

Doing It Twice

2008-04-14T01:39:00.000-07:00

As an Agile Coach I have found it somewhat disheartening the way Software Development Organisations choose to cherry pick Agile practices that fit in with there current beliefs and culture. I mentioned this in my "Me too Agile" post.T

Interestingly it is not the first time this has happened. After hearing several reports that Winstons Royce original paper on Managing the Development of Large Systems proposes a very different approach to software development then the way "Waterfall" is has been interpreted.

It turns out that Royce understood the importance of feedback and Emergent Design too:

STEP 3: DO IT TWICE After documentation, the second most important criterion for success revolves around whether the product is totally original.If the computer program in question is being developed for the first time, arrange matters so that the version finally delivered to the customer for operational deployment is actually the second version insofar as critical design/operational areas are concerned

If you replace the word documentation with communication then there is nothing here that would look out of place in a modern paper on Emergent Design. It is a shame that people chose to cherry pick back then. Taking the bits that were convenient and leaving the bit that weren't.

Skimming through the paper, there are a few eye openers that show that nothing is new in Software Development, we just choose not to listem. This was one of the more surprising ones that jumped out at me.

Architects - Who needs them?

2008-04-11T00:06:00.000-07:00

William as responded to my post on "Me too Agile" and his response made me think that my post required further explanation. An Architect as a software development role is a relatively new idea, appearing in the mid 90's along with the boom in 'OO' technologies and middle ware.

Before this the only architects I was aware of produced paper drawings for buildings. The architect would design, and builders would construct. The term Architect is borrowed from the building trade. Its roots in the building industry are quite revealing I believe and say a lot about the assumptions, philosophy and organisational cultures that led to its use in Software development.

It seems strange to have to say this, but in the early days of software development, there wasn't roles like Project Manager, Business Analyst, Systems Analyst, Architect, Test Manager etc. There was usually a couple of guys who sat down and decided that they wanted to write a computer program. When Thompson and Ritchie decided that they wanted to write the Unix Operating System they didn't have an Architect. They did the design and wrote the code themselves. They even went as far as producing their own programming language, so they created their own tools too. As a team they were self reliant, with a single focus the creation of a working program.

The purpose of my post was to question the relevance of the role of architect in an organisation that subscribes to Agile values. William in response quoted the Agile manifesto as a definitive statement of Agile Values. Well before the term Agile was coined, the category of software development methodologies we now call Agile went under the name light weight processes. What all these methodologies had in common was a rejection of an high ceremony approach to software development. Approaches like OMT, Objectory, RUP (which now coincidently claims to be (me too) Agile :)) that prescribe a number of distinct development phases with a number of distinct roles, handing off work to each other through a number of concrete intermediate work products. 'Light weight' meant getting rid of as much of this as possible and getting back to a way of working that would be more familiar to Thompson and Ritchie.

This view of the world, builds on a management philosophy that says that decision making should be delegated to the lowest level possible within an organisation. In short the people who do the work are best placed to make the right decisions. This idea of worker empowerment builds on the work of Edward Deming, and reaches a pinnacle in the Toyota Production System as described in the writings of Taiichi Ohno.

In keeping with this philosophy, the Japanese have developed a number of approaches to new product development, which we commonly refer to as Lean, Just-in-Time, Agile, etc. One of these approaches is deferring design decisions to the last responsible moment. The reason for this is to allow concurrent design. Toyota do not have Architects. They have a chief Engineer who champions the project and as an overall vision, but he does not make design decisions or tooling decisions. These decisions are delegated to the design teams who are autonomous. These teams try to keep their options open avoiding looking themselves into decisions which are not easily reversed later. The reason why they do this is because they do not own the whole design. For example the boot design team may need to change their design late into the development cycle. If the team responsible for the interior space of the car decide that people in the back really need an extra inch legroom to compete with the latest competing models in their target market segment. Then the boot design team need to be in a position to respond quickly, by changing their design reducing the boot space to accommodate the larger interior. This agility is why they defer irreversible design decisions.

Simon's paper on architecture, although it doesn't use the word "Agile" explicitly, makes much play of this Agile principle. The point I was making is that this principle is part of a wider philosophy, and that philosophy is not consistent with the idea of Architects and Architecture as borrowed from the building industry.

Taiichi Ohno would not recognise the use of deferred decision making in a context where the architect was the technical authority yet not responsible for doing the actual work. What we have here is a mix of two separate belief systems. The belief system that says that decision making should be centralised stems from the Management teaching of Fredrick Winslow Taylor and his idea of Scientific Management. A good example of this philosophy in practice was Henry Ford and the Ford Company. The Japanese came to the US after WWII and looked at the Ford production System and largely rejected it as inefficient and wasteful.

The point I'm making is that these are two separate contradicting belief systems, based on different values. We can question the relative merits of each, but cherry picking practices from one belief system and applying them out of context into an environment with the opposite belief system is not likely to produce the intended results. The beliefs come first and the organisational structure stems from those beliefs.

So whilst it is true that Simon did not appropriate the label Agile, he did attempt to reconcile the role of Architect which is essentially a taylorist construct, with the principles of Agile product Development which stem from the beliefs of Deming.

This is the background to the point I was trying to make.

Generics - An OO Anti-Pattern

2008-04-10T00:51:00.000-07:00

I'm obliged to use Java 1.5 at my latest client. One of my gripes with Java is that it doesn't encourage an OO programming style. As a Coach, I tend to find that most Java programmers lack a full understanding of OO design principles. In fact I can count on one hand the number of Java programmers I've come across who have an understanding of OO which is at least as good as mine.

In contrast all the Smalltalk programmers I've met, understand Objects very well and I'm sure most could teach me a thing or two. So Sun decided to revamp Java, a supposedly OO language. You would have assumed that they would have borrowed even more from Smalltalk, but no. Instead we get Generics. So why?

Before answering this question. I should really spell out why Generics are not compatible with OO design principles. Objects are meant to be loosely coupled runtime entities. Objects do not exist at compile time, they come into existence when you run your program (or with Smalltalk, they come into existence the moment you load your image). Objects should hide both their state and implementation from others, this is how they achieve low coupling. All that is exposed is their message interface. In Smalltalk, this interface is known as the Object's Protocol. So to communicate with an object, and get it to do something useful, you need to know its protocol and nothing else.

OK. Lets compare this with Generics. Firstly in the Java view of the world, message sends are replaced with virtual functional calls. A Function call as a way of sending a message to an Object is only possible in Java if you know the type of the receiver. In Java the receivers' Type is either a reference to its implementation (its Class) or to one of its implemented Interfaces. So straight away the idea of hiding knowledge of the implementation through message sends is lost. So along comes generics. Does it improve matters any? Well no, in fact it makes things worse, a lot worse.

If the answer (return value) to the message sent to the receiver is a collection, then the Type of Objects that it may contain is also part of the message interface in addition to the Type of the Collection (container) itself. So Generics leak a lot more information about containers. It gets worse when you consider subclassing and method overriding with generic containers. The complexities of what should represents a valid answer to the same message is mind boggling.

So generics leak information like a sieve, and break the basic OO tenant of information hiding (encapsulation) and low coupling. Information hiding is useful, because it allows you to substitute objects at runtime. The substitute could extend the protocol of the original, or implement the same protocol in new ways with different side effects. This idea is what is commonly known as polymorphism, and gave birth to the idea that OO programming could lead to components and re-use.

So back to the question Why? Information hiding is powerful when it comes to malleability and extensibility, but for some perhaps it is too powerful. In Smalltalk to know the Type of an object you need to send it a message. There is no other way. The Type is not manifest in the code. So the Smalltalk IDE is a running Smalltalk program containing a bunch of Objects. Classes themselves are Objects and to know what Type an Object is the Class Browser Object sends it a message to which the answer is the Objects Class. So with Smalltalk you only get to know anything about an object once it is running. Because of this the Smalltalk environment is always alive, always running, all the way through the programming cycle. The Smalltalk image, when loaded contains both IDE objects such as the Class Browser and developer Objects such as application Classes. The Browser sends messages to Class Objects to reveal their methods, and to method objects to reveal their source code. Programmers edit the code, and then send a message to the Compiler Object to compile the method. None of this is possible without running the image.

This approach doesn't help you if you don't like running the code to find out what it does. If you want to perform a static analysis you need more information at compile time. So using Generics is a way of providing this information removing the need for dynamic casts. So why would you want to get rid of casts? Casts are one of the gaps in your ability to fully analyse your program statically. A cast is an explicit admission that static analysis can't help in some scenarios, and you still need to defer some type checking until runtime. Generics is an attempt to bridge this gap, in an attempt to provide complete type safety at compile time.

For programmers who like their code to tell them what it will do before they run it, then this meta-data laden declarative approach is viewed as a benefit, but as Steve Yegge points out, programmers should not need such "training wheels". To know what a program does, what you should do is run it (test it). This unnecessary meta-data obfuscates the code and limits the degree to which the code can be deemed fully Object Orientated.

True Object Orientation relies on late-binding, which occurs at runtime. The whole point is that "you don't know for sure" what it is you are sending a message to. Allowing the receiver to be substituted. Manifestly stating that you know, limits polymorphism and artificially restricts the computational model.

Pattern Languages and Painting by Numbers

2008-04-03T23:27:00.000-07:00

Steve Yegges post on Noobs has really got me thinking about models and the purpose they serve. I think that models help fill a void in the design space. To begin with all we have is a problem and a blank canvas. A skilled artist will look at his subject and start sketching perhaps at first with a pencil. He will then refine and re-work his sketch before moving on with oils.

To the novice artist the blank canvas must be daunting. I have noticed that novice developers with a poor understanding of OO find OO programming from first principles a daunting task too. They are much more comfortable starting out with a template or a framework. A standard sketch. So if the subject is a web page, then the Struts framework which implements the model-2 web application pattern is a comforting place to start. All the novice then has to do is colour in the details. Just like painting by numbers.

When coaching, I have had an uncomfortable feeling about patterns. Novices are great at learning patterns by rote, not so good at knowing when to apply them. And even when they do apply them correctly, few seldom know how to tailor such patterns to create a novel design themselves. Worst still is where someone else has imposed a pattern and the programmer follows it religiously not noticing that the pattern doesn't quite fit for the problem at hand.

Artist do not paint by numbers. Sure they learn a lot of styles (patterns), but ultimately their creations are there own. So how do they create and come up with something novel when faced with a blank canvas? Well they rely on first principles. They understand perspective and how it works. They understand light and how it works too. They understand their medium: oils, canvas, brushes and strokes. They have a deep understanding of their subject aswell. Bone structure, muscles, skin tone etc. They also have a good eye for detail and can see things in their subject of interest, drawing out important nuances, and giving them prominence in their final piece of work.

Sound technical understanding based on first principles, a catalogue of prior work (patterns) for inspiration, a keen eye for significant details, and a well honed instinct for creativity. These are the quality of a good artist. They are also the qualities of a good programmer too.

I might be over ambitious, but I believe that average programmers can obtain this level of skill with coaching and practice, and model obsession as we have come to know it is not needed and can be replaced with something better. In a series of blogs I hope to strip away standard patterns and get back to first principles in an attempt to show the cost we are enduring by painting by numbers. I also want to show that going back to first principles isn't that daunting and can be both beneficial and a lot of fun.

Me too Agile

2008-04-03T01:55:00.000-07:00

William Martinez has pointed me to a paper which talks about Emergent Design from an Architects perspective. Apparently Architects are now deferring architectural decisions to the last responsible moment in an attempt to be more Agile.

On the same site they present a skills matrix defining the role of an architect. looking at the matrix it looks as though the new Agile Architect is a cross between a Systems Design Engineer a Senior Development Lead and an Agile Coach.

To me the term Agile Architect is a misnomer. Architecture as I understand it is about technology selection high level design. In an Agile team, the team are responsible for making such decisions. If the Architect is part and parcel of the team or is an outside consultant that the team can consult then I guess that this approach is still consistent with say Scrum. But ultimately the team must decide. Since the team is ultimately responsible for delivery. Anything else would brake the central tenant of Scrum which is that the team should be self organised and make their own decisions.

It puts me in mind of Scott Amblers website on Agile Modeling. Again a central tenant of Agile Development is feedback. Yet you don't get much feedback from a non-executable UML Model until pretty late in the development process. So again Agile Modeling is a bit of a misnomer.

So why are we seeing such things? Well in my opinion this is the consequence of organisations that have structured themselves around a waterfall "production line" view of software development who never the less like the sound of Agility, yet do not want to confront the need for organisational change. To me these are just symptoms of Agile as a fad. Wannabes jumping on to the bandwagon. Me too Agile.

True Agile seems to be taking root in environments with less cultural baggage such as new startups. But for those who are merely interested in trying out the latest fad without making a real commitment to cultural change, they can change their practices a tad and re-brand themselves as Agile. "Me too" Agile. Which is what I believe we are seeing here.

SOA and the N-tier Fallacy

2008-03-30T03:25:00.000-07:00

My last post on Emergent design got me thinking about other common architectural patterns (in addition to ORM) which may not necessarily be the best solution for the types of problems to which they are commonly applied.

I believe N-tier architecture where N=3 is a common choice, is yet another example of Big Upfront Design locking us into a unnecessary straight-jacket even before we look closely at our problem. Now the purpose of each tier in such an architecture is to provide a service interface to subsequent tiers. The goal being that tiers are loosely coupled and hence inter-changeable. From a clients perspective the tier you will most likely want to change is the presentation tier. So a PDA client will want a different presentation tier then a desktop client, for example. But both clients would want to share everything else from the service tier down.

All of this is common wisdom and as spawned a whole architectural vernacular of its own around the term Service Orientated Architecture. So what is wrong with SOA you ask? Well nothing, other than the term is not very specific and it tells you nothing about the type of problem to which it applies. SOA is the proverbial solution looking for a problem. So lets start this discussion again from the other direction. Lets choose a concrete problem and then identify a clean working solution.

The problem: My users want to store a bunch of related data records in a database. They also want to create, retrieve, update and delete (CRUD) these records remotely through a web browser interface .

OK. They want to store related data records. Sounds like a job for a relational database to me. They then want to view these records over HTTP. Well the verbs in HTTP (POST, GET, PUT, DELETE) sort of map to CRUD operations so why not make use of this? Also my user will view data records via HTML web forms presented to the browser. So there is inherent data coupling between my web presentation and my entity/domain model. Why not except this coupling as intrinsic?

From an Emergent design perspective, the simplest thing that can possibly work is an architecture that allows you to create data record hashmaps from HTML presentation views, and persist them in a relational database. Better still from an OO perspective, it would be even better if those hashmaps could persist themselves, and perhaps do other useful house keeping like data record validation that only they should know how to do. Viola - The simplest CRUD architecture possible. A Controller to handle the HTTP verbs and map them to actions on our database. An ActiveRecord that models database records as an hashmap and is capable of persisting itself, and a View that translates the model to an HTML presentation ready to be sent back to the browser over HTTP.

Sounds very much like Ruby on Rails to me. Now how do you think DHH got there? Did he use big upfront design, or did he choose to focus on his problem afresh and apply emergent design principles? Well it should come as no surprise to find out that Rails was harvested from a real word CRUD application that evolved over time. It should also come as no surprise either to find out that 37 Signals are big advocates of TDD. In fact Rails is the only web framework I know that has TDD built in.

One last thing. The astute amongst you are probably asking what happened to our nice separation of concerns and SOA? Well if you look at my problem statement I didn't need it (YAGNI). My client was using a web browser. I've saved myself a heck of a lot of unnecessary code and shipped my release 3 months early. Its now out there making a ROI for my customer whilst those N-tier folks are still deciding how many tiers they need. Hang-on I hear you cry. So what happens when your customer does decide that he needs another client to the system:

Problem2: Not only does the system need to support CRUD data entry by people, but in addition, we have recently gone into partnership with another company who wants to access the same data records using a remote system. So we now need some way to support this too.

The world as changed so we need to change our system to suite. Unlike the BUFD guys we will only make such an investment if driven to do so by our problem. OK. We understand how to map HTTP verbs to database actions, we also know how to map our model to HTML. Given this it makes a lot of sense to re-use this infrastructure and choose a service interface that makes the most of HTTP, we settle on a RESTful approach. Our previous CRUD based controllers still apply, but now the presentation needs to be machine readable XML instead of HTML. We could put a lot of if.. else.. logic in our controllers to determine what type of presentation to create, but we decide that this would represent a lot of code duplication, so we choose to encapsulate this logic in one place which all controllers can use. This approach is so successful that we then decide to factor it out as part of our home-grown framework. Sounds pretty much like how Rails 1.2 was born.

So now we have a Restful Application that is fully SOA aware and is built on our existing technology base. The only new technology is XML, and given that XHTML is a variant of XML, we could argue that we haven't had to introduce any new technologies at all. So we have managed to introduce a Service tier when we needed it at very little cost.

Some following this discussion may come to the conclusion that DHH just got lucky. To them my answer would be that you create your own luck. By gaining a deep understanding of the problem, keeping the design clean and simple and deferring unnecessary design decisions rather than prematurely locking yourself into false assumptions; then 9 out of 10 times evolutionary paths will present themselves without you having to plan (guess) for them upfront.

In the 1 out of 10 cases where there is no low cost option, then what your problem is telling you is that your application requires diverse personalities, and whether you plan for this upfront or evolve it later, producing such an application will be expensive. The benefit with emergent design is that by applying the YAGNI principle, you will not incur this cost unless you really need to.

Emergent Design

2008-03-27T12:58:00.000-07:00

William Martinez as an interesting repost to Cedric Beust's recent criticism of TDD. As always William as something interesting to say, and he makes the point I made on Cedric's blog, that TDD is no the whole design picture and you need to use a different design "language" at different levels of design abstraction.

Given that William is an Architect, I seized on the idea of using "different" design languages and what I see as the over use (misuse) of modeling languages such as UML. I tried to express this in a comment on his blog, but for one reason or the other (probably due to its length :^)) his blog wouldn't except it.

Thinking it through, I believe what is central to Cedric's views on TDD is a lack of awareness of Emergent Design and how it works. It would be interesting to see Williams take on Emergent Design since it gets rid of different design roles such as Architect, Analyst and Programmer combining them into one. The way it should be :)

Getting rid of Architects? Provocative I know. Anyway I include my comment here as a description of Emergent Design. A warning. It contains plenty of strong opinions :)

Hi William,

I sort of agree. We have spent a long time trying to place "good code" in a box. You can't legislate for "good code" yet we try. The design layers you speak of are one way of stratifying code into deferent levels of design abstraction in an attempt to legislate and box up Software Development. Yet it is the code that determines the correctness of a program and ultimately the success of the design. Once we arrive at the code, all other models are merely a projection of the one true model, the code model. Other models have no concrete existence.

So how do we test the quality of our code model? Well it is like the Shrodingers Cat paradox. We do not know whether our design is correct (works) until we execute the code. We do not know whether the cat is dead or alive until we look in the box.

I think an over emphasis on modelling can lead to a situation where we forget this basic truth. Second to working code, we would also like our code to be clean. How do we know our code is clean? Well for OO systems removing code duplication through inspection works well. TDD punctuates the coding process allowing you first to check for correctness through verifiable computation (a green bar), then inspect and safely remove code smells such as duplication, coupling, poor cohesion etc with the safety net of knowing that you aren't breaking correctness.

Using this approach, your design can emerge empirically. I agree that this bottom up approach to design has its limits. It can lead to locally maximised solutions (micro-design). So how do you find the global maxima (macro-design)?

XP practices come to our aid here too. By implementing the solution one customer prioritised story at a time, you are focused to extend your macro design incrementally, the simplest way that can possibly work. So at the story level your macro design is emergent too. But that leaves one last question. Each story's implementation needs an architectural context. How do you decide on your architecture? Well other than Kent Becks "Metaphor" idea XP provides little guidance here and we are back to guessing.

Unlike traditional design though, we acknowledge that our chosen architecture is just a guess and is yet untested by our requirements (stories). Given this we make a minimal investment in our untested architecture. Some people call this a walking skeleton. As we add flesh to the bones, we may find out that the bones are in the wrong place or are the wrong size, so we change them, and in this way our architecture emerges too. We may end up with the skeleton of a Horse that emerged from the skeleton of a Pig say, all driven by our requirements and an overriding drive for simplicity.

How do you describe this process in a book? The answer is you don't. This process requires a great deal of skill and judgment applied at each punctuated phase/cycle. Top designers like Kent Beck design this way, lesser designers can learn this approach by example and through experience and practice. The idea of learning from a Teacher and developing your skills through practice is not revolutionary and is the norm for most creative professions. Unfortunately we have managed to fool ourselves into thinking that programming is not a creative profession. So we do not have the teachers and coaches, and we do not provide newbies with the concrete feedback they need to learn from their experiences. If a programmer ends up having to code around your architecture and never gets an opportunity to communicate this to you, how will you learn and improve?

It is true that not all modelling can happen at the keyboard whilst writing tests/code. But it is possible for a lot more modelling to occur this way then people think. I would say that as much as 90% of design time can be spent at or near the keyboard, and the other 10% can be ephemeral (back of a napkin, white board session, CRC cards etc). I would also say, that modelling at all levels of abstraction should be done by the same person. Remember that code is merely a human readable representation of the computational model, and it is the computational model that is our final program.

So how do you get here? Well firstly it needs to be experienced to be believed. You end up with a design that actually fits your problem, rather than a design that you thought would fit your problem, and got locked into early on. Have you ever got to the end of a project and thought if I knew at the beginning what I know now I would have designed it completely differently? Well with emergent design you get to change the design every time you learn something new. No need to guess during a BUFD (Big Up Front Design) phase.

An example of BUFD (IMO) is how we all have got locked into ORM, when it may not necessarily be the best fit for the type of problems (CRUD) we are trying to solve. I think we have become model obsessed. Steve Yegge has an interesting blog post where he explores this further. I think Steve is right.

We need to stop aping other professions and spend more time mastering our own. Software Development is a unique craft that calls for a special set of skills that cross a number of layers of abstraction. Unlike the building trade it doesn't lend itself to stratification. We need to be able to program, design, architect and analyse all at the same time in small micro cycles each lasting as little as a few seconds. We need to punctuate the process with concrete feedback. At each cycle we need to focus on a specific problem one at a time. Our cognitive abilities are finite and we must divide and conquer, so as not to overload them. And we need to allow our design to emerge, deferring design decisions to the last responsible moment, where we have learned the most and are best equipped to make the right decision. We also need to recognise when to break out in the light of new information, and reconsider the big picture, either as an individual or as a team exercise.

These skills may not fit neatly into the stratified organisational structures of our current software development organisations. But the Uber programmer doesn't fit into this world either. We need to be producing more Uber programmers who are creative, highly skilled, and massively productive. We also need to develop organisational structures where Uber Programmers can thrive. Cooperating with other programmers and other parties like the customer; all working together collaboratively as a single team with a single goal: working software. Common code ownership, and using code as a design comunication tool is aligned with this view.

We need to be creating more Kent Becks.

Post Agile - In search of a new Metaphor

2008-03-17T05:57:00.000-07:00

Agile Software Development seems to be going the same way as a number of other fads in IT, such as Object Orientation and Design by Contract. I use the word fad, because of the faddish adoption curve. Firstly there are the few original proponents with a deep understanding of the new idea, then next is the herd who lack the depth of understanding, but are eager to try the new practices, which they learn by rote, parrot fashion, and then there is the demise and ridicule as the parrot fashion, "cookie cutter" adopters falter and the original idea is discarded, baby and bath water.

One of my favorite quotes is that "the label is not the thing" from the book "Language in Thought and Action" by the psychologist S.I Hayakawa. So Agile Development is just a label, not every team that chooses to use this label is doing the "thing" the original proponents had in mind. So what to do? Should we avoid labels completely in fear that they are abused? Thats one possible answer, but labels do serve a legitimate purpose. A label is a short hand that is useful in conveying a great deal of meaning. You wouldn't want to repeat the whole Agile Manifesto every time you wanted to describe projects that share its values and principals. So the label Agile is a valid one.

The problem I think is that the Agile label conjures up an image that doesn't convey the whole thing. The image is of Gazelles leaping freely through the African Bush. Swift movement, and free spirits operating without restraint.Does this image summarise the values and principles of the Agile Manifesto? Does it summarise why people do Agile development? On one level, the answer is a partial yes. From the perspective of a programmer like Kent Beck, I suppose Agile is an apt metaphor. The freedom to hustle, move quickly and use your skills to get where you need to be fast, without impediment. But does this metaphor summarise why companies like Toyota use an Agile approach? I don't think it does. For them a more powerful metaphor is Lean. This word conjures up an image of disciplined efficiency, where everything serves a purpose and waste is eliminated. No excess baggage, no fat. But does the Lean metaphor describe the whole thing?

I wonder what would have happened if the proponents of the Agile Manifesto had decided to use the word Lean instead? I suspect that board rooms would have picked up on "the thing" more readily. As its stands Agile is often misconstrued by both Managers and Developers as a collection of technical practices, even though tools and technical practices are de-emphasised in the Agile Manifesto in favor of Leadership and sound Management.

Lets look at some of the root ideas of the Agile Movement. Lets see what Edward Deming has to say (Taken from Mary Poppendiecks Leadership presentation on InfoQ):

Appreciate the System

A system wide view is fundamental -never sub-optimise
Manage the relationships between suppliers, producers and customers

Knowledge of Variation

Most variation is "common cause" and is inherent in the System
Trying to eliminate this variation only makes things worse
Systemic problems lie beyond the powers of the individual worker
To reduce variation, provide leadership in improving the System

Theory of Knowledge

Use the Scientific Method to measure and improve Systems

Psychology

When it comes to people, the things that make a difference are skill, pride, expertise, confidence and cooperation.

To me Deming sums it up well. The noticeable addition in my mind by Deming to the "science" of Management is to recognise that traditional science plays a very small role, and an appreciation of human psychology is far more important.

Perhaps this is why Agility defies precise scientific definition. You either get it or you don't. Agility is a statement about people and how they choose to collaborate. Agile skills are human powered, and are propagated by example by Leaders who understand the importance of empowering others. When talking to people who don't get it, I think I'll be avoiding the word Agile in future. Without a shared experience to pin it to, this abused label is bound to lead to confusion. For newbies perhaps I'll try the yet untainted label "Lean" instead?

On second thoughts, I think its best to avoid labels altogether and just talk about what I do and why I do it. I'm conscious that any label will inevitably fall short, because the "thing" is all encompassing. It is "Agile", "Lean", "Adaptive", "Human Powered" and... a bunch of other stuff too! Just like human pyschology the thing is multifaceted and not easily summarised.

I guess labels are great for marketing and propoganda, but not so great at conveying non-trivial ideas.

IS ORM a Dead End?

2008-02-09T03:08:00.000-08:00

I have never been over the moon with ORM. It solves the need to write SQL in your code, and to iterate through database results sets to form data structures, but it never really addressed the mismatch between true Objects and Data in my opinion.

Before going on to explain why, I would also like to say that ORMs also facilitate Domain Driven Design. To gain this benefit though, your application is no longer relational database centric. Instead your database becomes a pseudo object repository, storing object state and object graphs. This is fine for OO programmers but can look strange to those from a pure relational DBMS background. What DDD is really saying IMO is that OO driven application design calls for an OO database. ORM and a RDMS will do, but your DDD model is still OO not Relational.

So where does this leave us? Well hopefully acknowledging that the OO Model and the Relational Data Model are just two different models. Two different ways of modeling the world around us. Which model should we choose? Well it depends on what we are trying to model and why.

Bob Martin wrote an interesting blog post on the OO Model, ORM and the Active Record pattern. According to uncle Bob, the OO Model tries to model the world in a way that provides immunity to data changes. The idea is that data is hidden (encapsulated) inside objects, and that externally no one knows or depends on the objects data type. Instead you communicate with objects through messages. The data types can change as long as the messages and the objects answer to those messages (behavior) remains the same. This is the basic idea behind polymorphisms, which provides for immunity to implementation change, including changes to the encapsulated data type.

The relational model is very different however. Its goal is to allow you to find the data you want quickly. It does this by recognising the relationships between data types and using set based maths to "join" record sets. To do this, the relational model chooses to expose all entity data. The exact opposite of encapsulation. Exposed object (entity) data is used to join records and filter results sets during querying.

So how do we square the circle? Have a model where data types are hidden, but also where we can perform powerful queries? I don't think you can, and this is why IMO almost all ORM solutions end up exposing the underlying data (all those getters and setters). In a DDD application you learn to live with this ignoring the exposed data and augmenting the data mutation methods with true OO "business" methods that provide data encapsulating, "business domain" behaviour. But given that the data is exposed, there is nothing stopping others accessing the data themselves, breaking encapsulation and bypassing business rules. In fact if you intend to do queries you need exposed data to perform joins, filters etc.

So there is no squaring the circle, and your DDD isn't truely OO. What you have are data records that can also behave like objects, but due to the lack of encapsulation afforded to true objects and the opportunity this provides to violate OO semantics, you cannot say that your design is immune to data type changes. I believe that this is Uncle Bobs main complaint with the Active Record Pattern.

I turn the argument on its head. You are producing an application where you are interested in data types, where you want to display those data types to your users and where you want to explore relationships between data types. This is what we would call a classic database application. In these applications immunity to data type changes is an impossibility. You cannot hide data types, because data is what your user is interested in. Your user is also interested in some behaviour which express business rules, but most of those rules have to do with maintaining data integrity and enforcing relationships between data entities. Your user wants to view his data.

In such an application OO encapsulation amongst domain objects serves very little purpose. Polymorphism is only useful as a means of grouping data types with common attributes, but not as a means of grouping 'objects' with common behavior, and data encapsulation becomes meaningless. So why not forget about objects and data encapsulation and use exposed mutable data types instead? Well functional languages have been using this approach for years, an hashmap (Dictionary) with name/value pairs is a mutable data type. You can represent any data type you like by nesting hashmaps. Accepting that all data will be exposed, and that data types are likely to change is a much better fit for database applications where users want to store, navigate and query data.

Given the hashmap as the primary abstraction, where does behaviour fit in? Well there is data agnostic behaviour such as "Create", "Retrieve", "Update", and "Delete" which applies to all data types. In addition to basic CRUD is querying behaviour like "Select From", "Join" and "Where condition". These are all data type agnostic and could be provided by a framework like LINQ or Rails ActiveRecord. Then there is data type specific behaviour like "Age" which calculates the age of any data type that contains the attribute "date of birth". This would need to be provided by the application programmer and associated with a set of data types.

This is why I think that Bob got it wrong in his conclusion. Data is King in data centric applications (in contrast to behaviour being King). The Active Record pattern as implemented in Rails acknowledges this fact and treats domain entities as pseudo Objects and doesn't try to pretend that they are proper objects with encapsulation. LINQ takes the same approach too.

For database applications where set based data queries are important, then ORM has always been a misnomer in my view. What we have really been doing is data-structure relational mapping. With Rails and LINQ we are now moving into "dictionary relational mapping", which in my opinion is a more natural way to model data centric applications then "ORM".

Monopolies, Cartels and Developer Choice

2007-11-03T02:54:00.001-07:00

This recent article on the serverside has got me thinking (yet again) about the state of the software industry as a whole. The article is about a blog entry that asks Why are Sun and Microsoft now clambering onto the Ruby bandwagon? And can Ruby be the next Visual Basic?

These questions expose some interesting assumptions: A) That there is a gap (or perhaps a gulf) in the Software Development market left by VB6 that still needs filling. B) That this gap can only be filled by a language that is backed by a major vendor like Sun or Microsoft. The last of these assumptions appear to be true when you look at the demise of Dolphin Smalltalk. Dolphin Smalltalk was an ideal OO replacement for VB6. Yet it perished. Why?

Thinking about these questions raised an interesting thought. I have long concluded that Managers/Strategist and Architects make technology purchase decisions not programmers. Managers are of course risk adverse and like to stay near to the technology herd. This makes sense given that they often know little about technology themselves. Like the saying goes, no one ever got fired for buying IBM. So vendors have learnt over the years how best to sell to Managers and decision makers. As for Programmers, well they are an easier proposition. If the Managers have bought into a given technology and the herd is clearly moving in a given direction then the programmers will clamber to get the "new" technology on to their resumes.

These cultural patterns have traditionally led to monopolies such as with IBM and then Microsoft. With the rise of Java and J2EE we have witnessed yet another phenomena, that of the cartel. In response to Microsoft's near monopoly, a group of companies decided to gang together to form a cartel. On the face of it J2EE is standards based, but until very recently it has been the cartel (Sun, IBM, Oracle, BEA etc) that have dictated the overall direction. In so doing mainstream developers are left choosing between either Java/J2EE or C#/.Net. This boils down to practically no choice at all.

So how will things be in the future? Well the cartel have split up the J2EE cake amongst themselves along database lines. But it is not just the J2EE market that is split this way. the whole serverside cake seems to be split along database lines. The thought that came to me is that Oracle and DB2 shops tend to use Java/J2EE and Microsoft SQL Server shops tend to use .Net.

So the key to sales seems to be the database. This makes sense, because from a management perspective, "the database" is the companies most important technology asset. Well actually the important asset is the corporate data not the database server, but people seldom seem to make this distinction :^). So after spending almost eight years contracting and 17 years in the industry what have I witnessed? Well the database salesman comes in and sells the managers a database. On top of that they then convince the managers of the need for middle ware. If the database is DB2 then the natural choice is Websphere, Oracle then OAS, SQLServer then .Net etc. They then convince them that it is a good thing to mandate a single technology stack for everything, because having a "corporate strategy" is important. So all teams end up using DB2 and Websphere whether the application requirements and timescales merit it or not.

Into this mix has landed open source. Some parts of the open source community are aping this traditional vendor behavior. So JBoss and Red Hat Linux see themselves as a traditional server side technology company. Others are just programmers interested in programming, and sharing tools and languages that help them become more effective at delivering. But wait a minute, how can programmer centric technology succeed if it is the managers who get to make all the decisions?

Well if the managers are themselves Programmers then this approach works very nicely. Paul Graham has been blogging on this very topic for quite some time, and the Agile community worked this one out some time ago too. Everything we do needs to be programmer centric, because at the end of the day it is the quality of the code and the productivity of programmers that counts the most. Organisations that take this approach are massively productive when compared to traditional software development organizations.

So it seems to me that I need to find work with people that realise the importance of programming and understand that they don't need Oracle (Or IBM or Microsoft) to look after their precious data. Freed from management ignorance, I'll get to choose the best tools for the job, and who knows programming could even become fun again :^). It sounds like I need to be looking for work with a startup!

Ruby, Smalltalk and Lisp Revisited

2007-10-14T02:55:00.000-07:00

In my last post I suggested that Ruby may have some advantages over Smalltalk, especially when it comes to borrowing some of the more powerful features of Lisp. First thing to say is titling the post "Ruby versus Smalltalk" wasn't the cleverest thing to do. The responses tended to contain more heat then light. I hold up may hands and take full responsibility for this. Given the amount of FUD that Smalltalk has suffered over the years, it is hardly surprising that the Smalltalk community are protective.

I've been recently looking closely at Rubinius, a Ruby implementation that uses a Smalltalk/Squeak-like VM. Rubinius borrows many of the architectural ideas of Smalltalk/Squeak and is Ruby implemented mostly in Ruby. Eventually even the VM will be implemented in a C-like variant of Ruby called Cuby, much like how the Squeak VM is implemented in Slang.

I'm pretty taken by Rubinius. The approach taken is to implement Ruby right, and in so doing they've gone back to Smalltalk. The other thing that is very impressive is that all the Ruby libraries in Rubinius are implemented using BDD. So there are Rspec specifications for all library methods. The full Ruby library is not complete yet, but using Rspec is an excellent way to finally specify the Ruby language and in so doing allow lots of people (like myself :^)) to get involved in the Rubinous project.

So back to Lisp. The Rubinius compiler generates a simple Lisp (s-expressions) as an intermediate form on the way to byte code. There are lost of good reasons to do this, many of which are dicussed here:

http://on-ruby.blogspot.com/2007/01/will-rubinius-be-acceptable-lisp.html

Another reason not mentioned is that Lisp can become a common intermediate form allowing several source languages to interoperate. So Ruby or Smalltalk could be compiled to Lisp then Lisp to byte code. Peter Frisk has already implemented a Smalltalk to Lisp parser in Vista Smalltalk so this is possible. It also fits in nicely with the language orientated programming ideas that Martin Folwer has been touting for quite some time now.

Rubinius is gaining a lot of momentum in the Ruby community. So why is Lispyness important? I think the answer is flexibility and allowing change. Something that Alan Kay said seems relevant here. The first Smalltalk implementastions where built with Lisp I believe, and whilst Smalltalk was still within the confines of Xerox Parc the language was being experiemented with and changed all the time. Alan Kay complains that once Smalltalk became public in 1983, the language became fixed. He had hoped that Smalltalk-80 would be redundnant by now, and that it would have ben replaced by something better.

The Ruby community seem up for evolution, whilst also being a very practical bunch. The Rubinius project is cooperating with the JRuby project to specify a common Ruby across all implementations, something that I believe was a problem within the Smalltalk comunity. By considering how best to bring Lisp to Ruby they are also showing an openess to change. Matz seems open to change too, Ruby 2.0 although not revolutionary, does show a willingness to deprecate old features and break backwards compatibility, in the same spirit as pre Smalltalk-80 Smalltalks.

I still think that Ruby is closer to Lisp then Smalltalk, but that doesn't stop Smalltalk inlining Lisp or adoptng more Lisp like features if it wants to. In a sense the two languages share a common root. The difference seems to be history and the two communities. The Ruby community seem to be more willing to experiement and evolve, whilst the Smalltalk community is still holding onto Smalltalk-80 as the common Smalltalk dialect. I guess the question is whether the Ruby community can evolve together and stay as one, avoiding the fragmentation that occurred with Smalltalk.

From what I've seen on the web, there is plenty of reason for optimism. If the Rubinius/JRuby people pull it off we may end up in a situation where we don't need to choose. We can mix Ruby, Smalltalk and Lisp as and when appropriate.

Ruby versus Smalltalk

2007-08-03T05:30:00.000-07:00

Is Ruby a lesser Smalltalk? Well I use to think so, but now, after using Ruby for a while, I'm not so sure. Well Smalltalk definitely excels when it comes to tools, but as with Java, Ruby's increasing success will mean that better Ruby tools are sure to appear soon.

The reason why I'm not so sure is the ability to define higher order functions in Ruby, in much the same way you would with Lisp. For those that don't know, a higher order function is a function that writes functions, so code that writes code. Rails uses this technique all over the place. For example the famous scaffold method is a second order function. Your Controller class writes itself when it gets defined at runtime. AFAIK this just isn't possible in Smalltalk or is it? Why is this important? Because you can add new control structures to your language, extending the language and creating DSLs in much the same way you can with Lisp macros.

It looks as though Ruby is a true successor to Lisp in a way that Smalltalk never has been. Is this true or am I missing something?

Open Source Software Businesses

2007-07-04T06:23:00.000-07:00

Following on from my last post, that was motivated by this discussion on TSS. In answer to the question. What motivates Gavin King when he is selling Seam? Well after some mis-communication, Bill Burke of JBoss was very honest about this. Jboss is a Software Business, they make their money mostly through support subscriptions. So JBoss sell support licenses for a Software Product, The JBoss Application Server. This is the business that Red Hat bought.

So is there anything wrong with this? Well no if you're open and transparent about it. I definitely do not work for free, and as Bill rightly points out, much of what JBoss as achieved in the open source arena would not have been possible if they hadn't grown the business. Also, one size doesn't fit all, so although J2EE5 isn't the simplest way to build a web application, as some of the comments to my last post rightly point out, in some cases J2EE5 is a good option.

So where does that leave us? Well I still feel that describing JBoss as "community driven" like they do on their website isn't completely "accurate" :^). JBoss like any other business will do what it sees to be in it's own interest. So putting back on my cynical hat. Gavin Kings motivation when selling Seam? Create an opportunity to sell more JBoss Subscriptions perhaps?

Again, fine - but does this qualify him as a "community leader"? In my opinion no.

The J2EE King is Dead - Long live the J2EE King!

2007-07-01T03:28:00.000-07:00

I've quit posting on Objects. Gilad Bracha is much better qualified then I and besides he has a cool blog. Gilad reckons that Java is not OO. Well that's something coming from the guy who led development on the Java VM for a number of years.

As for me, I've decided to take Gilads advice and ensure that my next project is not a Java one. I'm seriously getting into Ruby and Rails, and I'm also dabbling with Flex. I'm finding the Ruby forums kind of boring though, basically filled of helpful knowledgeable people, supplying useful technical tit-bits, without an axe to grind insight.

In contrast the Java forums like TSS are a lot more "fun". I've noticed a trend lately, the OSS projects that exposed the swindle known as the J2EE Application Server are now seizing the J2EE crown for themselves. Spring, which is a neat framework, that basically helps to overcome a Java language smell (Gilad discusses this too), now sees itself as the new J2EE King. I have read several articles from Rod Johnson, and whilst he is very eloquent in describing the pitfalls in J2EE and in particular EJB, at no point does he mention that IoC is merely a pattern to deal wit the fact that object construction cannot be done in an OO fashion with Java. Now is this just an oversight on his part, or is he just simply ignorant of this fact?

Also, Johnson seems to have got himself into a squabble with the JBoss/Hibernate guys. The JBoss camp has an alternative vision for the new J2EE, and they see themselves as the true successors to the J2EE throne. Again, at the core is a framework to deal with the limits of the Java language. Hibernate with the use of CGLIB manages to overcome the static nature of Java, by late-binding POJOs to a database agnostic persistence mechanism at runtime. Has it escaped Gavin Kings notice that Rails manages to do the same thing, within the Ruby language and without having to resort to ugly tricks?

You would have guessed that both of these clever guys would have realised that the problem is with the language, and that they would be spending their time building OO frameworks using a more appropriate, 'OO' programming language, but they aren't, so why?

Well the simple answer is money. Java is an established market, and they both intend to exploit it. Unlike the old guard, which they have successfully over thrown, these new upstarts have an open source business model. I'm not an expert on OSS, but apparently you can make money out of software by giving it away. The thing about these new pretenders however is that in my opinion, they are both fundamentally dishonest. I don’t believe either of them is being totally open about their true motives. When an Oracle salesman walks up to you in a pin-stripped suite and offers to sell you a product, you know what you’re potentially getting into, but when Gavin King offers you Seam as the solution to all your problems what are his motives? Is he truly altruistic ?

Marc McFluery of JBoss fame is a very rich guy. So how did he make his money? The JBoss camp has chosen to stick close to the JCP and build on Java "standards". Their latest offering in this vain is Seam 2.0, which builds on JSF1.? and EJB3.0. As I understand it JBoss with Seam is fully J2EE 5 compliant. So I guess this is what Red Hat thought they bought. McFluery must of laughed all the way to the bank!

So back to Gavin and his Seam2.0. Without going into details, Seam tries to do with J2EE5 what Rails does with Ruby. Now I guess you're saying that a full blown J2EE5 App Server is not needed to do what Rails does, even with a limited language like Java - yes, true, but... JBoss is J2EE5 standards complaint so they must use all the J2EE5 standards :^).

From all accounts Seam at its core is an elegant framework, the problem is though is that it solves the wrong problem. The problem it should be solving is how do I build web applications with Java quickly? (to which the obvious answer is, you don't :^)). Instead the problem it chooses to solve is how do I build web applications using J2EE5, including EJB3.0, JSF and a full blown App Server? If you sell Application Servers, or product consulting for Application Servers like JBoss does, then perhaps this latter problem does need a solution. So back to the question, what are Gavin Kings motives when he is selling Seam? Well if you are as cynical as me, you would say that it is to push JBoss as an Application Server and to sell more product consulting.

These political and business issues have dogged the IT industry for a very long time, yet they are never adequately debated amongst the developer community in my view. My concern is that there will be a lot of young naive developers out there that believe that they can trust the likes of Rod Johnson and Gavin King as honest “community” leaders, and will actually invest hours learning either Spring or Seam, only to find out that it all was one big con and that their time would have been much better spent learning Ruby.

Anyway, like I say I intend to be doing less Java in the future. Personally, I prefer not to identify with a specific programming language and I like to see myself as just a Programmer first and foremost; that way I’m free to choose any programming language that suites me. My advice to the Java community though, if they are listening, is to watch out for the new pretenders. They could turn out to be worst then the ‘leaders’ we have now.

Look Mom, No versions.

2007-04-10T01:57:00.000-07:00

Something that as troubled me a little, is the link between 'blue' OOP and Agile practices. I say 'troubled' because Agile development is widely misunderstood and I fear that 'blue' OOP is widely misunderstood for similar reasons. It is not widely acknowledged, but Agile development practices stem from OO programming practices and Smalltalk. Agile development breaks with the traditional idea that Software development can ever be deterministic. The desire for a deterministic software development process stems partially I believe from the desire for control. And the thing that non-Agile Managers want to control the most is change. We have all heard of the cost of change curve, where the cost of software change increases by a factor of ten, the later in the development process the change occurs. With a functional or procedural language this makes sense. Since everything is by default inter-connected, then any change is likely to propagate to other areas of the code. So the last thing you want to do is to 'allow change' once you have established a significant code base.

On the surface, this anti-change policy sounds like common sense. It even sounds similar to other common sense ideas like ' get it right first time', which seemingly promote an early determination of requirements and design upfront. But if you think about it, not allowing changes once you have code rules out maintenance, and also assumes that your customer knows exactly what he or she wants at the outset.

In practice it turns out that this 'big up front design' approach just doesn't work that well, and in fact exploring and experimenting in code, and allowing change to occur all the time can work much better. This is the philosophy behind dynamic languages, where feedback from experimentation can be gained relatively quickly. The problem though is that the 'embrace change' approach is counter to what most of us have been taught, and is not widely seen as 'industrial strength'.

The other thing about embracing change is that your code base needs to support change. The existence of unit tests and the 'safety' that they provide has been well documented in the Agile world. Tests do allow you to detect bugs introduced by changes, but it would be much better to avoid introducing bugs in the first place. This is where OOP and encapsulation comes in. If you can encapsulate your business logic inside objects that only communicate through loose messages, then the chances are that you will be able to introduce changes which remain encapsulated themselves and do not spread throughout your code. So OOP allows you to introduce changes cheaply, by reducing coupling through message sends and increasing cohesion through encapsulation. The biological metaphor of a cell which Alan Kay uses is a good one in this regard. Cells are independent and autonomous and go about their daily function independent of each other. The cell membrane provides encapsulation, so changes that occur within the membrane are localised and specific.

Using this technique people like Jeff Sutherland (Scrum) and Kent Beck (XP) realised that you could flatten the cost of change curve, and thus introduce changes at anytime during the project development life-cycle. This simple realisation opens up a number of new possibilities.

Gilad Bracha has a presentation where he takes this idea of allowing continuous change to it's logical extreme. Gilads idea is of bits that rot. Software that automatically updates itself continuously whilst running. In such a world version numbers have little meaning. The only version that counts is the latest. Without a fixed version your software is no longer an artifact, but a service consisting of a continuous stream of improvements and fixes. Old code rots away and is replaced by new code continuously. This idea fits well with a biological metaphor too.

Take a look at the presentation by Gilad. It is almost an hour, but is well worth watching.

Deep into the Blue - Industry Titbits

2007-03-30T02:09:00.000-07:00

I have found the responses to my blog thus far a bit intriguing. The general response has been a stiff defence of the current status quo. My opinion (and it is just an opinion), is that the status quo isn't really delivering, and we could all be doing a lot better.

A few articles I've come across recently have re-inforced this opinion. The first article builds on my view that late-bound OO message sends can form the basis for language interoperability. Peter Frisk as recently implemented high performance 3D web rendering using Smalltalk. The usual response to using Smalltalk for such a CPU intensive application is that Smalltalk is too slow. So how does Peter do it?

Well, Peter has utilised the layered DSL idea I've discussed before. So the primitive 3D graphic rendering is performed in ActionScript, which as I understand it is a static, high performance, compiled OO language which runs on the Adobe Flash runtime (Virtual Machine). On top of this he layers a Lisp interpreter, which allows you to call ActionScript primitives from Lisp. On top of Lisp he then implements a DSL that so happens to be Smalltalk-80. As I understand it the Smalltalk implementation is fully interpreted, but this doesn't matter, because the bulk of the graphics rendering is delegated to ActionScript. BTW a domain language programmer using Smalltalk, doesn't need to understand action script at all. Pretty impressive. Take a look (requires Flash 9).

It may look like Peter has gone to all this work for nothing. After all it can all be done in ActionScript, so why Lisp and Smalltalk? The thing is though, is that Peter appreciates the power of late-binding. Smalltalk components written in this way can be mashed-up together to create new objects, in the same way that people are using html and java script to create mashups on the web today.

Another titbit that I have come across that was interesting is a post by Gilad Bracha. Gilad is famous for his work on the Java JVM and worked for Sun until very recently. For me Gilads most impressive work was performed before he joined Sun over 10 years ago, when he did research on Smalltalk, Mixins and Traits, which eventually lead to Strongtalk, the high performance Smalltalk implementation with optional manifest type annotations and a static type checking system. I've discussed Strongtalk before. Gilad has been talking about Self and the idea of slots. C# has the idea of properties, which is a way of implementing getters and setters as used in Java. What if you just make the variable public? And later you want to change it to a method? In both Java and C#, this could mean changing a significant amount of code. With Self this isn't the case (with Smalltalk you can't make an instance variable public anyway, because it breaks encapsulation, so the problem only exists for subclasses). Gilads blog has some interesting examples of better ways to solve/avoid common programming problems using late-bound languages.

Finally, Croquet has announced the release of version 1.0, and is no longer in beta. At the same time the Croquet consortium was officially launched. The consortium is a body to promote the development and adoption of Croquet. Along with a number of Universities the Consortium also contains Hewlett Packard and a new Start-up: Qwaq, a commercial company that will focus solely on collaborative applications using Croquet.

There seems to be growing momentum in the blue plane. Peter Frisk and Vista Smalltalk is definitely worth watching, along with Croquet. I also see Strongtalk as promising, not so much for it's superior performance, but as a bridge into late-bound programming for programmers who are reluctant to relinquish their preference for manifest type annotations and static type checking.

Deep into the Blue with Croquet

2007-03-19T02:35:00.000-07:00

Time to look forward. My last couple of blogs on Object technology have focused on the perceived benefits of the current crop of incumbent main stream OO languages. We explored a bit of history and got a bit bogged down IMO over the subject of Type Safety and Program Correctness.

If anything I think the discussion demonstrated the point that we still don't know how to write safe programs with any degree of certainty, and that any program is as good as the programmers who produced it. So for me the term "Type Safety" is a bit of an oxymoron, because being type safe doesn't infer program 'safety' at all!

Accepting that there is no guarantees, perhaps we should let go of the pink past and explore the new blue OO idea a bit further. To do this we need to take a pure OO approach, with scant regard for incumbent technology. Croquet is a project that chooses to look at software Engineering afresh from a pure OO perspective. The question posed by Croquet is:

If we were to start again, and build an Operating System with modern assumptions about computer power, what could we do today?

To this question the Croquet team have come up with some answers:

A VM that works bit identical on all platforms. They achieve this by writing the Squeak Smalltalk VM in a subset of Smalltalk itself, called Slang.
Given bit identical behaviour, replicate objects across the web, with the guarantee that replicated objects will behave bit identically.
Using Object Replication, and synchronised message sends, create a shared virtual Time and Space, across the web, they call this TeaTime.
Use Peer-to-Peer communications to remove the bottle neck of centralised servers.
Late-binding to ensure that the system can grow and change organically. Also allow non Croquet components to be consumed into the Croquet world.

I will explore Croquet in detail over the next few blogs. Here is an article which is a excellent primer on Croquet for the uninitiated. It is difficult describing Croquet, because like the Sony Walkman, Croquet is something new and innovative, and unlike anything we have seen before. The closest description to the vision held out by Croquet is the virtual computer world presented in the movie "The Matrix".

Croquet is The Matrix.

Type safety, An Oxymoron?

2007-03-09T05:12:00.000-08:00

I think I've found a concise definition for type safety. I found it on the C2 wiki, which is a great source for programming related info. Anyway here it is:

Type Safe
Any declared variable will always reference an object of either that type or a subtype of that type.

A more general definition is that no operation will be applied to a variable of a wrong type. There are additionally two flavors of type safety: static and dynamic. If you say that a program is type safe, then you are commenting on static type safety. That is, the program will not have type errors when it runs. You can also say that a language or language implementation is type safe, which is a comment on dynamic type safety. Such a language or implementation will halt before attempting any invalid operation.

Taking the first sentence. This rules out any type of conversion so int->float is type unsafe, it rules out any type of dynamic cast too. So that basically rules out C, C++, Java and C# as type safe. Moving on to the main paragraph we see that aswell as static type safety there is also the concept of dynamic type safety. Using this as our bench mark, still rules out C and C++, but deems Java and C# to be dynamically type safe (if we choose to ignore the issues surrounding conversions of primitives of course). This laxer definition of type safety also includes languages like Smalltalk, Python and Ruby. So all modern OO languages are dynamically type safe.

If this is true, what is the dynamic versus static typing debate all about? Is type safety an oxymoron? Reading on further on the C2 wiki:

There are various degrees of type safety.
This is different from TypeChecking.
See also StronglyTypedWithoutLoopholes, which is another term for (at least) dynamic type safety.
CategoryLanguageTyping

So using the "degrees of type safety" argument, Java could be said to be more type safe then say Smalltalk. This kind of makes sense, since even though Java is not fully static type safe, it is partially so. So type safety is relative. So you can rate languages on their degree of type safety. Statically typed languages are more type safe then dynamically typed languages generally. If you click on the link CategoryLanguageTyping you will find out that what we usually refer to as static typing isn't actually called static typing at all, the proper name is Manifest Typing, Static Typing means something else and includes Type Inference. Given the common use of the term static typing, I have chosen up to now not to use the proper term which is in fact Manifest Typng.

So what does all this buy us? At best we are partially type safe if we choose to use a language like Java. Partially? Is that useful? Either I'm safe or I'm not right? For example, when releasing to production, I can't tell the QA Manager that I believe my program is partially safe. He wants to know whether my program is safe.

So how do I know that my program is Safe? Well simple, I test it!

I could go into strong versus weak typing and the consequences, but the links are there if you're interested. No program is Type Safe, and to claim so is a bit of an oxymoron. IMO typing is no substitute for well thought out tests, but type checks can help to detect and track down bugs (either at compile time or runtime). Where I believe manifest typing is useful is in improving the readability of code, improving the comprehension of large systems, and improving tooling support for code browsing and editing. Examples of this is the code completion and refactoring features in Eclipse. Smalltalk has these features too, but with manifest type annotations, tools have that much more information to work with.

The downside of manifest typing is that all type systems use 'structural types'. Structural types are based on the structure of your code. Depending on the code annotations available, manifest structural types can limit expressiveness. This is why languages like Scala have invented a more expressive set of type annotations, to overcome the type constraints imposed by languages like Java. Strongtalk's type annotations are even more expressive. This had to be the case because the Strongtalk type annotations had to be applied to the existing Smalltalk 'blue book' library, and this was originally written without any manifest type constraints whatsoever. The other downside of manifest types is that your code is more verbose.

So ideally what you want is :

* Manifest type annotations that can express intent and do not constrain you (or no type annotations at all or type inference)
* Strongly typed without loop holes at runtime
* Tests that tell you whether your code is safe.

Type safe doesn't exist, and partial type safety is a poor substitute for the above!

"Sorry, you're not my Type!"

2007-03-07T05:35:00.000-08:00

Before we explore the future potential with Blue OOP, I thought it only fair to address the perceived advantages of pink OOP first. After all, I have labelled pink OOP as just an extension of the "old thing", but who says that the old thing was all that bad? Was there anything about the "old thing" worth holding onto?

The old thing I am referring to is C. C was one of the first 3rd generation languages to be used to write an Operating System for micro-computers (I think?). That Operating System was Unix. Prior to C most micro-processor OSes were written in assembly. I mention microcomputers, as this is/was the name for computers built using micro-processors. Prior to the micro-processor computers where huge boxes of electronics built from discrete components.

Early microelectronics placed considerable constraints on computer software. Many of the Computer languages used on big "mainframe" computers just weren't suitable for microcomputers especially personal computers. Outside research organisations, personal computers had very little processing power and very little memory.

The success of C was largely due to the success of Unix. Unix, was ported to a wide range of computer systems. Also, with C you could get very close to the efficiency of assembly language, and unlike assembly language your code was portable.

This is a longer introduction than I had hoped, but a lot of people have forgotten this history and it is useful to remind ourselves of it. So by the early 80's C was the personal computer language of choice.

Then along came Objects. So the challenge was how to bring OOP to PC's and still retain the efficiency of C. There were two candidate languages, both derivatives of C. These were C++ and Objective C. C++ replaced the message passing of Smalltalk with a virtual function call. This ensured that method dispatches would be as efficient as possible. The downside is that C++ is an early bound langauge as binding to concrete methods occurs at compile-time. Objective C however, chose to retain message sends, this means that Objective C is late bound, but as a consequence is less efficient at method dispatch then C++.

Given the hardware restraints at the time, the majority of the industry went with C++. The only PC company I know of that went with Objective-C was Steve Job's Next with their NextStep OS.

So the big advantage of pink OOP is efficiency. As time has moved on however, some in the industry have tried to re-write history and claim that the big advantage of pink OOP is type safety. Now I must admit, I do not know exactly what type safety means. There are a few things that I do know however:

* A Class is not a Type
* Late/early binding and Static type checking are orthogonal concerns
* Static typing is usually associated with early binding
* Static typing can be applied to a late-bound dynamic language like Smalltalk.

The first bullet is a conceptual flaw in C++, that Java attempts to solve by introducing Interfaces. The problem with Interfaces though is that they are selective. So sometimes in Java you bind to a Type and at other times you bind to an Implementation (Class), an unsatisfactory compromise IMO.

I'm going to get myself up to speed on "type safety". My experience has shown that static typing as used in languages like C++ and Java can greatly reduce the expressiveness of the language. So instead of the compiler being my friend, it ends up being a straight jacket, stopping me doing what I know would work best, if only I was allowed.

This is just opinion of course. I have come across one static type system that I believe will allow me to have full flexibility. This is a Typechecking system for Smalltalk called Strongtalk. Here is a link to a paper on The Strongtalk Type checking System. The current Strongtalk is slightly different to the description in this paper. If you are interested in the difference you will need to look in the Strongtalk documentation in the Strongtalk download bundle. I believe Scala is an attempt to bring more expressiveness to static typing on the JVM so I will be taking a more detailed look at Scala too.

It should make a neat comparison. Two static OO type systems one targetting a late bound langauge (Smalltalk), the other targetting an early bound language (Java), it will be interesting to see how they compare.

BTW. If there is anyone out there who can answer the question: What is type-safety? I would be more then happy to hear from you.

Revised 07/03/2007: Modified to acknowledge the role of Unix in the rise in popularity of C - Thanx Steve.

What Colour do you like your Objects? Pink or Blue?

2007-03-04T13:45:00.000-08:00

It's late and it's a Sunday, but I thought I'd just make a quick post to clarify a few things. What is OOP? Since Alan Kay's team coined the term 'Object Orientated' with the release of Smalltalk to the world in the early 80's OOP has become one of the most exploited marketing terms in programming.

It would be interesting to see when the term was first used. It wouldn't surprise me if the first published use of the term was in the original Byte Magazine article on Smalltalk in August 1981. So OOP was born with Smalltalk. Before Smalltalk Simula extended Algol to allow data structures to contain function pointers, but this was seen as an extension of data abstraction, and the term OOP wasn't used.

In Alan Kay's keynote speech at OOPSLA in 1997 he talks about a blue plane and a pink plane. The pink plane represents ideas which are an incremental improvement of existing ideas. The blue plane which runs orthogonal to the pink represents revolutionary ideas that break the old way of doing things, setting you off in a new direction.

Since the creation of C++, OOP has born these two identities. Firstly a pink identity, where OOP is seen as an extension of the existing thing, this was the view of Bjarne Stroustrup and what lead to C++ and ultimately Java. Secondly there is a blue identity, where OOP is seen as a new thing, which breaks with the old and has new requirements all of it's own. This second identity is most closely associated with Smalltalk and Self. It has also influenced other OO languages like CLOS, Ruby and Python.

These two identities so happen to deal with types differently, and the difference between the two is often referred to as static versus dynamic, but in truth, this dichotomy is a false one. The difference runs much deeper. The real difference between the two stems from their goals and their vision.

The C++ goal was to introduce OOP like constructs to C in an efficient way. To do this Stroustrup avoided the garbage collection, byte code, VM and late-binding of Smalltalk and went back to the much simpler and efficient model presented by Simula. The strength with this approach is that C++ is very efficient, the downside is that C++ is decidedly pink.

Self built on the platform of Smalltalk in an attempt to push further into the blue plane. The goals of Self were:

* Objects that are tangible just like physical objects in the real world
* Objects that share uniformity, just like physical objects do (everything is an object)
* Objects that exhibit liveliness, removing the modal nature of programming (no edit/build/run cycle)

All these goals are characteristics of Smalltalk, but Self wanted to take these characteristics much further, creating a fully graphical programming experience, where objects could be handled and manipulated from a visual palette, just like physical objects in the real world.

You can see that this 'blue' vision is very different from the pink one. One of the most obvious consequences is that with Smalltalk and Self there is no difference between graphical objects on the users desktop, and 'programmable objects' in the programmers IDE. In a sense the desktop becomes the IDE and the IDE becomes the desktop. Following from this the distinction between programmer and end user starts to blur. Also the distinction between object and application disappears all together. Each object is an application in it's own right, even the benign Number object '1' or '3' is an entity that can be manipulated at runtime through it's own GUI. The VM contains a large collection of such objects and becomes more than just a runtime, it becomes a Graphical Operating System.

Infact the object instance '1' is more than just an application. It also encapsulates it's own server with it's own synchronous message queue and it's own virtual processor. Adopting this semantic view of OOP means the runtime is now analogous to a NOS (Networked Operating System) spanning several virtual processing nodes. This is the semantic goal of blue OOP and why Alan Kay used the analogy of the encapsulated biological 'cell' in his keynote speech. I will expand on this blue OOP vision in a future post. But as you can see pink OOP is very different from blue OOP and the difference has very little to do with types.

Revised 06/03/07: Replaced 'Real Objects' with 'Physical Objects' in line with the terminology used by the Self team - Thanx Isaac

Programming Languages - Follow the leader

2007-03-04T03:48:00.000-08:00

A short interlude from my series of posts on Objects. Steve's last comment got me thinking. Why are some programming languages more popular than others? It would be easy to put it all down to cynical marketing by vendors, but that can’t be the whole story. It was this sentence in particular that got me thinking:

It may sound neat to allow developers to modify the language, but having used Smalltalk for more than 20 years, I have had to deal with the chaos that can result when different developers modifications conflict. I would rather have a controlled and organised process.

So an ordered and controlled process is seen as desirable. Ok but controlled by who exactly? The truth is that most people feel more comfortable being lead. I can wax lyrical about the technical superiority of languages like Self, Smalltalk and Lisp as compared to lesser languages like Java and C# (and even Ruby and Python), but this doesn't matter a jot if people just aren't 'comfortable' with these supposedly 'superior' languages.

With Java there is minimal degrees of freedom. If you want to iterate, there is one (non-deprecated) way. Want a call back there is one way. You do things the 'Gosling way'. It is all pre-packaged and rather assuring. I must admit when I first used java I found it's simplicity re-assuring too. It was definitely welcomed after the explosion of constructs that accompanied the transition from C to C++. C# has used the same formula, after all it worked for Java. Java has taken "the shrink-wrapped approach" further, beyond the base language. The whole J2EE application stack was supposed to result in "one way" to build enterprise applications. Reducing software development to painting by numbers.

This all works up until the point where the 'one way', just isn't the best way for you. What do you do then? Well you live with it, like the EJB community did for years, or you jump to something better suited, like pico-container or Spring.

Many Java developers are now jumping to Ruby and Rails for precisely the same reason. For many web apps, the full J2EE stack even with Spring and Hibernate, is just seen as overkill. Interestingly though, very few have moved to Squeak and Seaside, and even fewer to Lisp. Why?

Well in Matz and David Heineimeier Ruby and Rails respectively, have strong leaders. Benign dictators that prescribe "how things should be done". Ruby developers can model themselves on the approaches recommended by these leaders. Better still these leaders are developers, themselves, so there is an instant bond of trust. The Python community has demonstrated this phenomena even more so, with a single all knowing leader Guido van Rossum. Rossum even dictates how code should be laid out and how tab spaces should be used!

So in contrast how do languages like Lisp and Smalltalk compare? Well let’s start with Lisp. I like to think of Lisp as a Meta-language; a programming language for writing other programming languages. A good example of this can be seen at the Vista Smalltalk blog. Peter Frisk is using Lisp to build a Smalltalk interpreter on top of Flex. So as far as Lisp is concerned, Smalltalk is just a DSL, created using Lisp macros.

With Lisp you deal with fundamentals. The smallest construct in Lisp is called an atom. An atom is a single character and you can combine atoms to produce symbols, and symbols to produce s-expressions (lists) etc, all the way up to a full class hierarchy of objects and associated functions. You can even determine how s-expressions are evaluated with Lisp macros, so basically you can do what you like!

This power puts a great deal of control and responsibility in the hands of the programmer. Of course there are established patterns to help guide you, but there is no benign dictator making a bunch of design choices upfront. You have to make your design decisions yourself. You are on your own!

Some people will revel in this power and flexibility. Others though, are likely to find it daunting! Smalltalk follows Lisps lead, but provides a lot more pre-defined structure. It has a small syntax, just like lisp, and like lisp has a meta-model built to support meta-classes, classes, and object instances. Unlike lisp though, all objects interact through message passing and are fully encapsulated. Many objects in Smalltalk are part of the language, such as the Context object used as a stack frame, Block closure object used as a lambda expression, and compiler objects used to turn strings into byte code. So Smalltalk gives you a lot of structure.

Smalltalk wears it's heart on it's sleeve. With Smalltalk all this structure is written in Smalltalk, so as a programmer you can change any part of it as you see fit. So this is fantastic if you want to create your own specific Smalltalk dialect. But if you do, Dan Ingalls or Adele Goldberg won't be there to help you out. And you won’t be able to turn to the Smalltalk-80 "Blue Book" either. You will be in the same camp as the Lispers, on your own!

When I first came across Smalltalk I saw all the dialects as a concern. All these semi-compatible versions surely can't be a good idea? As I have become more experienced as a programmer though, I have come to see diversity as a good thing. Two analogies come to mind. The first one is biological. In nature animals ensure that there is sufficient diversity in the gene pool. Each individual is not a clone of all the others, so if a sudden virus attacks, some of the species will be wiped out, but hopefully, others will have immunity, so the species as a whole survives. I think Smalltalk has this strength. Depending on what is important, there is a variant of Smalltalk to fit the bill, and if there isn't, a dialect can be readily mutated to meet the need (in most cases). Languages that can’t adapt in this way, face the risk of dying out through natural selection (something I believe Java is in danger of).

The other analogy is spoken language. Spoken language is a living and changing thing. We do not speak the same way today as we spoke 300 years ago. Also we have regional dialects, a Scoucer for example, sounds very different to a Cockney, yet they both claim to speak English (the Queens English, not US English :^)).

In their own domains Scoucers and Cockneys get on fine speaking their own dialect. But in situations where they may have to communicate with each other, like with written English, they both fall back to 'Standard English". For Smalltalk, "Smalltalk-80" is the equivalent of Standard English.

So that's the language landscape as I see it from a cultural perspective. Where I think I agree with Steve, is that change is slow in software because of a number of reasons, many of which are cultural. Where I believe things are inevitably heading though is into a pluralistic world containing many languages and dialects, but also sharing a common base, a lingua-franca. I see the lingua-franca as being based on late-binding and message passing, but I’ll save a detailed discussion of this for a later blog. In this new world I see many domains with leadership being dispersed across them, and with several individuals taking a leadership role at different times and in different circumstances.

For this to occur, developers will need to be more comfortable taking the lead themselves, and getting rid of the "training wheels". Technically, there are tools on the horizon that could help here, protecting the less self-assured. I see Language workbenches as described by Martin Fowler as perhaps helping here. A language workbench could provide a reassuring wall between the meta-language and the domain specific language, providing reassurance and safety for domain language programmers.

Supporting tools aside, with the rise of open source and open source languages, I believe there is strong evidence of this cultural change happening already! I see this change as inevitable as the industry grows up and matures.

Objects - I know that already!

2007-03-03T01:19:00.000-08:00

I recently received an email from an old adversary from TSS (The Server Side). Steve and I are kind of friends now - which is nice considering that we have never met, and only know each other through posts on TSS, e-mail and through our blogs.

Anyone who follows the news threads on TSS knows that I can be pretty vociferous with my opinions about Objects and the shortcomings of Java. Well I've infuriated Steve on many occasions, leading to long exchanges... One of Steve’s pet peeves, is me continuingly quoting Alan Kay. So you can imagine my surprise when Steve sent me this link to a keynote speech given by Alan Kay at OOPSLA in 1997

BTW, For anyone interested in Object technology, there are a whole set of videos available on the web showing the history of Objects and the primary players involved going back to the 1950s.

Steve is an ardent Java supporter, and I had posted a link to this same video and several others, many months ago in an attempt to cure him of this unfortunate affliction :^) Well many months later he stumbled across the same video himself, and he wanted to discuss it with me. Steve has over 20 years software experience (a fact that he is fond of sharing :^)), and has used several OO languages over the years including Smalltalk. So what was there to discuss prompted by a 10 year old video by Alan Kay?

Well you can all judge for yourselves. I would urge any programmer to watch this video. It deals with fundamental programming concepts, which most of us have dispelled from our consciousness long ago. Why? Because we know it already! We all know what an operating system looks like. We know what professional, industrial strength programs look like too. And we all know an "enterprise strength" programming tool (language + IDE) when we see one! We've all used/seen Eclipse, IntelliJ and Visual Studio. All of these tools are marketed as 'Object Orientated', and all of them are supposedly state of the art!

If you look a little closer though, and peel off the shiny veneer from these tools, underneath they look remarkably like 'C', 'vi’, 'make' and ‘cc’. Not much has changed since C/Unix in the 1970s. We still use the same old while loops and if statements, still the same edit/build/run cycle. If a C/Unix programmer had been put in a time capsule in 1977, and re-awakened today he would find tools like Java and Eclipse pretty familiar and would be up and running with them in days.

So why has so little changed in 30 years? Here is an explanation I've lifted from an Article by Dafydd Rees on Croquet and Squeak:

"Kay blames this lack of innovation on the fact that most adults employ instrumental reasoning to evaluate and apply new ideas. This means that adults have difficulty evaluating new ideas because they're carrying too many existing goals, and too much context to be able to see the full potential of new ideas."

One of the beauties of children is that they are untainted by our pre-conceptions. Each new generation looks at the world afresh, with new eyes, and kids perennially ask the question why?

My plan is that this post will be the first in a series, where I will be questioning strongly held assumptions about object technology. Hopefully Steve will comment too (apparently his epiphany was only short lived!). Free from marketing and spin; the idea is to have a useful exchange on where we've been with objects, where we could/should have been and were we should go next.

Like Alan Kay says: "The Computer Revolution hasn't happened yet".

If you are genuinely interested in Object technology; in a language neutral sense; then book mark this blog. It should be interesting and your input is welcomed.

Java, Objects and Static Types

2007-02-12T04:03:00.000-08:00

I seem to be on a roll myth busting. This one is prompted by a comment by Peter Kriens, in response to my previous post on Java and component models. The sentences that drew my attention where "interestingly, the type information in the language allows us to provide quite a few guarantees." and "for large systems where you get legacy parts from all over the place there is something to say for type information ..."

Now it would be only fair to allow Peter to explain himself further and not to read to much into what he has said, but his words did bring to mind a common myth, namely that dynamic languages have no type information, and perform no type checking. Well they do!

I'm old enough to remember programming in C, with no stack trace and the dreaded "Segmentation fault - Core dump" Unix message on program failure. C allowed no type information at runtime. You could declare pointers of type (void *), which meant that they could point to anything you liked. You could cast to an assumed type and the language would not check for you - in the end you would try to access a segment of memory not allocated to you by the System and bang!

Here is where my memory gets a bit hazy, but I believe things improved with C++. There was still void* pointers, but I believe C++ introduced a dynamic cast that did some checking at runtime.

So a long intro, but I wanted to get everyone on to the same page. So dynamic languages like Smalltalk do their type checking at runtime. If a type mismatch is detected, the program doesn't go bang like with C, but the language notifies you of the error and will even launch a debugger at the appropriate line of code. So Smalltalk is a strongly typed language, the difference between it and say C++ is that the type checking occurs at runtime not compile time (Smalltalk also gets rid of void* pointers, Java does likewise).

So what are the consequences of these differences?

* Well with Smalltalk, all type mismatch errors can only be detected by running your programming. So no coincidence then that test driven development and SUnit came from the Smalltalk community. With C++ the compiler at compile-time performs checks statically. So some believe that unit testing is less critical (I disagree).

* With Smalltalk, the overhead of dynamic message dispatch and the runtime checks makes the language inherently slow. With C++ there are few runtime checks, all function call indirection is removed at compile time and hard coded into a VTable. This allows for the use of an optimising static compiler, which inherently produces faster runtime code.

* Smalltalk is fully polymorphic at runtime. What this means is that objects can take many forms (implementations) at runtime. So any class of object that satisfies a caller’s request can be substituted in at runtime. This is known as late-binding, and has significant implications. C++ is only partially polymorphic at compile-time, common interfaces must share a common implementation (A common base class or abstract base class). At runtime object type is fixed (hard-coded), so no polymorphism.

So along comes Java. Without justification, I will assert that Sun produced Java in a hurry. Oak was aimed at low powered, low memory home devices, so the static optimising compiler route was the obvious one to take. So Java has inherited many of the properties of C++. Namely, highly performant and static (fixed).

For Objects that come 'alive', you require different properties. Sun did a lot of research in this area. The Self Project determined that Objects needed to poses a number of properties:

* Directness (you manipulate objects directly)
* Uniformity (everything is an object)
* Liveliness (modeless, no run/edit mode, fully dynamic)

These properties are inherent in Smalltalk, and the Self Project hoped to build on them. Self explored a number of important OO concepts that are still relevant today, but the implementation was a memory hog, and you needed a super fast computer to run Self.

Here is a good video on Self.

So the reason for saying all of this is that "compile-time' type safety was always an after thought with Java. The real reason why Java is static is performance. Also from a Marketing point of view the static label was useful, as the bulk of the developer community out there were C++ programmers and comfortable with the static programming approach. To them liveliness meant nothing.

In 1995 some of the original Self team were ready to launch a new Smalltalk implementation known as Strongtalk. The idea was to address what was seen by many as the shortfalls of Smalltalk - slow performance and no compile time type checking. They addressed the performance problem by optimising out dynamic method dispatch at runtime using a dynamic compiller. The approach was similar to that used in the JVM JIT today, but their approach also allowed for de-optimisation on the fly back to interpreted code, allowing them to satisfy the semantics of dynamic dispatch needed for 'liveliness'.

They also looked at the runtime type checking issue. At first they made the same mistake as C++/Oak etc and assumed that compile-time type checking had to do with the runtime implementation. It doesn't. Type declarations in a static language can be thought of as annotations. They annotate the code and tell the compiler how best to optimise method calls. They also allow the compiler to perform checks. What the Strongtalk team did was to delegate the optimising role to a dynamic runtime and retain the static type checking at compile time. To do this they needed to add Type declarations to the Smalltalk source code, this is done by way of annotation. So Strong talk can compile both Type annotated code and un-annotated code. At runtime Strongtalk maintains full dynamic messaging semantics - so you end up with the best of both worlds.

As the startup company who built Strongtalk in secret where about to launch it on the world, Sun came along and bought them up. The Strongtalk developers where moved over to work on Java and the JVM and are responsible for the Java hotspot JIT technology we have today. Type-feedback and Type annotations sat on the shelf for over 10 years.

Fortunately Sun as recently released the Strongtalk code as Open Source:

Strongtalk Home Page

But imagine how things would have been different if they had released Strongtalk as a product in 1995? Or better still if they had ported Java to Strongtalk using Type annotations?

* There would be no OSGi (no need)
* Ruby probably would not have blossomed beyond the Perl community (no need).
* Smalltalk would perhaps still be thriving today.

Paul.

Java, Component Models and Class Loader Hell

2007-02-11T03:29:00.000-08:00

I've been drawn back to TSS again. There is an interesting thread where Bill Burke from JBoss-Seam goes head to head with Rod Johnson of Spring. Now it doesn't surprise me that Rod Johnson is a bit of an egocentric jerk - I know someone who worked him for years who told me as much. It was interesting though to see the JBoss guys break their usual aloof pseudo-intellectual demean a: read here.

Both of these frameworks claim to provide a component model for Java objects. So the first thing to clear up is what is a component? Well as far as I can tell, a component is a course grained object that can be bound to other coarse grained objects after compile time. So components are an invention for static OO languages, in dynamic OO, all objects are components.

Right with that out the way (anyone who is still not convinced, I would suggest exploring the runtime differences between a virtual function call as used in C++/Java/C# etc and polymorphic message sends as used in Smalltalk, Phython, Ruby etc), here is why do I believe that EJB's are a flawed component model.

Let’s take a simple example. Component A uses Component B. A configures B and registers a number of callbacks with B so that B can notify A. A is packaged in its own EAR (or 'war' or 'ejb-jar') and so is B. So what is the problem?

Well A must have access to the Interface of B, I will call this B' and B must also have access to the callback interface of A, which I will call A'. So there is a mutual dependency between A' and B'. With the J2EE class loader model, each component has it's own class loader (CL). Class loaders are arranged hierarchically. So the CL for B can be a dependent of the CL for A, or visa-versa, but two class loaders cannot be mutually dependent.

So you cannot have A<-->B relationships between J2EE components. Apparently OSGi is meant to be addressing this, but it seems a pretty fundamental language flaw to me. It will be interesting to see how they plan to get over this one!

Oh, yes I forgot Spring. Well IoC is nothing new. It allows you to separate interface from implementation, deferring binding untill runtime (back to message sends versus virtual function calls again). So again, by default all dynamic OO objects have this property. In Java it can be achieved with the use of an Interface and the reflection API, which is effectively what the Spring bean factory does. Fortunately Spring doesn’t go in for this separate class loader nonsense - so no class loader hell here - but if you choose to package your Spring application into a WAR, then you have no runtime binding of components either, so A<-->B, with Spring becomes AB. So if you want to change implementation to A<--->C at deploy time, where C implements B' you can't.

So it turns out that Spring isn't a component model at all - oh dear!

So neither of these so-called "Component frameworks" can do what Smalltalk did out of the box, over 30 years ago. Maybe they should have discussed this poignant fact on TSS, instead of squabbling like school girls.

Paul.

Java - State of Denial

2007-02-04T03:45:00.000-08:00

I've been participating in an interesting discussion on Tuple Spaces over on TSS

Actually the discussion was on JavaSpaces, but since Tuple Spaces is the "non-branded" name for the architectural pattern - we quickly moved on to Tuple Spaces. There were some useful posts by people knowledgeable in the "distributed" Java technology space - like Cameron Purdy of Tangosol, but the debate quickly deteriorated (IMO) into a discussion of what you should and should not do with a Tuple Space.

To me this seemed very odd, because Tuple Spaces as a concept has very few restrictions, in fact the API is very simple, just four verbs (read, write, notify and take). So why the big discussion about what you can and can't do? Then it dawned on me. Some of us were talking architecture, whilst others where still talking implementation. So what Cameron was perhaps saying was that the Java implementations of Tuple Spaces have various limitations. Given that Java and distribution is Cameron’s area of expertise, he could well be right.

Yet, I know of at least one other implementation, not in Java that isn’t limited in the way he describes. So why would someone with his level of expertise not know of other implementations too?

I think this is a symptom of something I've seen descend over the java community over the years. I think what has gone on with the Bush Administration over Iraq, in some ways resembles what has happened to the Java community:

The first similarity is having all the answers. The JCP was meant to be the answer to everything. By committee Java would flourish and evolve. All good ideas would emanate from the JCP, and we didn't need to look elsewhere - Baker/Hamilton, no thanks!

The second similarity is staying the course. What do you mean that AppServers and EJBs are dead and buried? We intend to stay the course despite the fact that no one is adopting EJB3.0.

The third is denial. Java is still cutting edge, we are leading the vanguard in technology innovation - we will win through!

The Java community has become somewhat inward looking and entrenched IMO. Much of the interesting ideas are coming from outside Java, but the Java community consumed in its own grandeur is blissfully unaware of the rest of the world.

The sad thing is, that even Microsoft is evolving. WPF and WPF/e will deliver on the promise of Applets, which Sun made years ago. In contrast the Java community has stagnated, stuck with a language designed in a hurry to power toasters. Nothing is wrong with that btw, we've all got to start somewhere, the problem is though, that through denial no one sees the need for change.

So C# 3.0 is adopting even more from functional languages, whilst Java's thinks about closures. Ruby comes along and shows what can be done with a dynamic message send, and the Java community, slays the messenger Bruce Tate. The last time I looked Bruce was hiding out on the IBM forums writing articles for people who where interested in "crossing borders" like he had.

So the experts in distributed data solutions in Java haven't heard of Croquet. I guess it is not of interest, even though the Croquet team lead is the guy that invented Objects in the first place!

I don't believe Java is going to go away any time soon, but unless this culture of denial changes I can see it suffering a long lingering death. That in itself is not necessarily a bad thing, times move on. The problem though is that a lot of programmers have identified themselves with "Java". So they are no longer just programmers, they are "Java" Programmers. I once responded to a post that was clearly antagonistic towards Ruby from someone who openly admitted that he had only "read about it". Given his ignorance, I was surprised at his antagonism. When I asked him why, he came out with all the usual static typing arguments, and he also mentioned that he would not go on to a Ruby forum and make comments on Ruby. I guess by implication, he meant that I should not comment on a Java forum because I was obviuosly a Ruby programmer. I guess it never crossed his mind that I may program in both!

I could imagine Rumsfield coming out with a stupid statement like that. So the Java community has become tribal and partisan - and even Microsoft is more open minded and outward looking knowadays (have you noticed that XAML looks a lot like Adobe Flex?).

I guess the final service Java can provide to the developer community is as a repository for developers with a closed mindset. People able to think for themselves can just leave, much like Bruce Tate did, and move on to more interesting things. And the "I've just put the latest JCP acronym on my CV" programmer can stay with Java safely out of everyone’s way!

Paul.

REST Epiphany

2006-10-15T09:56:00.000-07:00

Ever since first hearing about REST, I have have been attracted to its' simplicity. Let's face it who wants to use SOAP and WS-*? I've struggled with REST though. What do you mean no new verbs? OO programming is built on verbs right? What would objects be without verbs (process)?

The I had my epiphany....

There was an article on REST on InfoQ that got me thinking about it once more. The article had a link to an application written in the REST style. The application allowed you to make telephone calls by sending messages using REST. It suddenly clicked. A RESTful request is a message send in an OO sense. OO is all about sending messages, it is the reciever that then interprets the message and selects an associated process depending on the message REPRESENTATION. For human consumption the message normally contains verbs, but as far as the recviever is concerned, the message is just a label (noun) which is used to select a process defined up front.

OO isn't about verbs, its about messages (nouns) which recievers act upon. So REST is kind of like a dynamic OO message send. An Example:

Buying a book at Amazon.

I would need to GET the book details and price info from Amazon, and
determine that indeed this is the book I want and that I like the price.

I would then POST my credit card details and delivery address, along with
any other relevant details back to Amazon..

As long as Amazon and I agree on the REPRESENTATION of these messages then
everything should all work fine.

Know using a keyword message format like in smalltalk I could send to Amazon the message:

Amazon.bookDetailsAndPrice: mybook.

As Mark Baker points out. A message in distributed programming is normally "a document with an operation and address wrapped around it". REST isn't like that. The operation is implicit in the URL and the REPRESENTATION of the document. So the same "message" in REST is:

GET Amazon/bookDetailsAndPrice?book=mybook
Response: A document with the book details an price.

And I would post back my credit card details:

POST Amazon/checkout

my credit card details
mybook
my address

Response: A document showing the success or failure of the transaction.

So you can send any message you like with REST, by transfering documents. It is that simple. So why didn't I get it before?

Paul.

Order and Chaos and the Power/Threat of the unknown

2005-12-27T07:34:00.000-08:00

Right as you can tell by the title this blog is about a number of ideas that I feel are related. I'll try and keep it short. What got me blogging was a post on the XP discussion group on yahoo. The discussion was about the newest XP practice "Slack" as mention in XPE2 (XP Explained, 2nd Edition). The question in most of the posts was "what is it?"

The answer was that no one really knew. The one person who should of known (Kent Beck) contributed a link to a forum discussing his fore mentioned book (XPE2), but offered no detail explanation of Slack himself.

I pointed out what seemed obvious to me, which was slack (leaving space in the plan) will mean different things to different teams at different times. So trying to elevate this behavior to a defined "Practice" was a futile exercise. Unfortunately only one person explicitly agreed with me. But I'm sure a few others did too, but chose not to be as vocal.

So why is it that we feel the need to define the indefinable? We are always modeling the world and trying to put it in a box. I think for many of us, that is just the way our brain works (left sided thinkers), Mathematicians, scientists and programmers.

Creative people (poets, artists, writers etc), seem to suffer from this need a lot less. To them something that is inherently indefinable is a source of fascination and interest. They tend not to find such things threatening at all. Concepts such as beauty and elegance resist precise definition, yet many communities quite happily share a common understanding of what these things are. This understanding becomes what is known as culture.

A book that had a profound effect on me was "Language in Thought and Action" by Awakawa. The book describes how thought is intimately linked to language, and how our choice of language can limit our thinking. In the book there is the idea that the label is not the thing. The map is not the territory. We use 'labels' in language all the time as synonyms for ideas, concepts, groups, activities etc. The synonym itself takes on it's own meaning and inference yet any instance of a thing that we choose to refer to using a synonym may have it's own unique characteristics which do not neatly fit into the meaning inferred through the synonym .

So the key is to know when your are using a label, and that the real world is a lot more complex and diverse. An example of this that comes to mind and is related to software is the CMM. The CMM is an attempt to model software development organisations. It identifys Key process Areas (KPA's) and their relationships with each other. Each KPA is assigned to a role that must exhibit concrete abilities.

As part of my training as a certified CMM assessor, I was exposed to a computer model of the CMM. It was great you could see the effect of changing abilities within a given KPA on other KPA's and on the process as a whole. Logically the model made a lot of sense. The big missing gap though was that people and organisations often defy logic.

So within the CMM a group of demonstrable KPA's represented an organisation who's output was repeatable on a per team basis (Level 2), an addtional group of KPA's defined an organisation whose output was repeatable and consistent across the organisation (Level 3). The next level of maturity, level4 , contained KPA's that demonstrated that an organisation could quantitatively measure it's performance as a pre-requisite to quantitative process improvement.

Unfortunately, the moment you expose this simplified model to real world situations it immediately begins to fall apart. For instance is it not possible that an organisation may have a single team that have demonstrably delivered repeatedly, using quantitative techniques. In such a scenario, the set of abilities demonsrated may cut across maturity levels, they may even cut through KPA's, leaving the CMM model in taters.

In a sense XP has fallen into the same trap as the CMM. When first introduced to XP I was told that you needed to do all the XP Practices together. I was told that they are all related and inter-dependent on each other. So after 4 years real world experience, XP has been shown to have gaps. Despite doing all the practices, there are still issues and problems that XP doesn't address. Is this surprising? So what is the solution? Well for Kent Beck it was add one more XP value (respect) and an additional practice (Slack).

The model is flawed, so just add to the model. There is another way. That is to remember that the model is just that a model. The real world is a lot more complex and diverse. The label is not the thing. The map is not the territory.

I could blog (rant) some more. But in summary, as developers we need to get better at using the other side of our brain. Not knowing shouldn't be threatening, a state that should be extinguished quickly, with ever more elaborate models. No, not knowing is just part of the rich fabric of life. In fact there is power in the unknown. In acknowledging that there isn't a simple formula, order can be found in chaos. Examples of this exist in philosophy, especially eastern philosophy. So Zen sayings like "Knowing without knowing" really do have meaning. Higher order thinking can help us to gain understanding in an inherently complex and chaotic world. All we need is the courage to embrace the power of the unknown.

XPDAY5 - Getting Customers to Know Themselves

2005-12-10T04:15:00.000-08:00

In the same session on "Getting to know your customer" mentioned in my previous post, Steve and Andy went on to describe Innovation Games. The goal of these games is to get your customer to think about their problems in a different way, and to come up with new innovative ideas on how to tackle them.

Interestingly, just talking to customers about what they want isn't always adequate. Often customers do not know what they want and end up telling you what they think you want to hear. I think this could be related to the three basic assumptions challenged by the Cynefin sense making framework (see my previous post). If our customers think that we expect them to be rational, intent-driven and ordered then they will answer our questions in kind.

By game play people are let of the hook. They do not need to pretend to be rational. They are free to get in touch with themselves and the true nature of their problem, after all it's just a game.

One of the games is called Product Box. Customers are encouraged to decorate a box containing the "product". The decoration should highlight the most exciting and sellabe features. The idea is to make the product appealing. Another game is called Speed Boat. Here customers are encouraged to draw a spead boat which is tied to things that are holding it back, things that are stopping them speeding ahead in their work.

These games are similar to the domestic probes used by Bill Gaver and fit in well with the Cynefin probe-sense-respond appoach.

Here is a link to more Innovation Games: http://www.enthiosys.com/innovation.php

The combination of Cynefin sense making, Innovation Games and Bill Gaver's Ludic Technologies really made XPDay for me. I'll never look at innovation and product ideas in the same way again.

XPDAY5 - Knowing your Customer

2005-12-10T03:49:00.000-08:00

One of the sessions at XPDay that I attended was about knowing your customer. Steve Freeman and Andy Pols introduced use to the Cynefin sense making Framework. This framework challenges three of the basic assumptions we generally make when trying to understand our customers and their organisations. The three assumptions are:

The assumption of rational choice - People will do what is rational.
The assumption of intent - Everything that happens is done intentionally.
The assumption of order - Organisations are well ordered, rule based entities.

It turns out that if you remove these assumptions, then most organisations turn out to be highly complex and chaotic (sounds familiar?). The framework goes on to suggest two approaches to effecting change in such organisations:

probe-sense-respond - for complex organisations and,
act-sense-respond - for chaotic ones.

This is very different to the sense-categorize-respond approach that has been traditionally taken (for example Business Process Engineering). They point out that the traditional approach is best suited to ordered organisations.

Here is a link to their paper:

http://www.research.ibm.com/journal/sj/423/kurtz.pdf

A bit of a heavy read. Thankfully the approach they recommend for complex/chaotic organisations, namely probe/act-sense-respond is already widely known as suck-it-and-see :^).

XPDAY5 - Anti-Intellectualism and Dead Fish

2005-12-10T02:25:00.000-08:00

Tim Listers keynote at XPDay this year was an impassioned plea for people to wake up and use their brains. An example of "brain-less" practice that he described was a team that continued to provide effort estimates to two decimal places even though their estimates where consistently between 100-200% out. Anyone who knows anything about maths (Tim's father) could clearly see the inconsistency between the estimating accuracy and precision.

So how does this type of thing happen? How do supposedly intelligent people keep making such fndamental mistakes? Organisational culture has a lot to do with it, coupled with the inherent difficulty of accurately predicting software development. The stats for software development failures are abysmal. Most projects fail and those that don't are either late or deliver less functionality then promised. Only a small percentage actually succeed.

Tim spoke about anti-intellectualism. People not wanting to know. I've experienced this too. If you propose an intellectually sound approach to tackling a complex question often the response is negative. It is almost as though people do not want to think. Worst still, it is though they are scared that if they did think things through, that they could come up with answers that aren't palatable.

The consequence of this willful ignorance is what Tim Lister refers to as the dead fish. Dead fish projects are projects where everyone knows that they are going to fail, but no one actually comes out and says it. The dead fish stays there smelling and everyone blistfully ignores it. Tim said that you can sense the mood in Dead fish teams, there is an atmosphere, a stench that is perceivable straight away. People know when they've been set up to fail, and it shows in their body language and general lack of enthusiasm.

Tim did a good job at identifying the problem, but didn't offer solutions. My experience is that telling people that there is a dead fish doesn't always help. Usually by that time, many people feel implicated and vunerable. Hearing the problem described so eloquently though was powerful. It has definately motivated me to do more about dead fish in the future.

XPDAY 5 - "Gathering" Requirements

2005-12-04T04:21:00.000-08:00

For me there was a sort of theme to XPDay 5. The theme was Innovation. How do we know that we're building the write things? Both of the Keynote speakers touched on this subject. It is an area that as concerned me too (see my post on software as an art). As developers we have managed to lull ourselves into the belief that product requirements are just out there and all we need to do is "gather" them. In Tim Listers keynote he showed a slide with a picture of a little girl picking a flower with the caption "Oh what a pretty requirement, let me gather it". It had us all laughing, but it made the point well.

Bill Gaver, the second keynote speaker spoke about his work with Ludic technologies. His interest is computational devices for the home. He believes that computers in the home need not be practical, work oriented devices, but instead can enrich the experience of domestic living in other non-practical ways. So the question is what should we build for the home? Well it turns out that the obvious approach isn't necessarily the best. Just asking people what they want in their homes doesn't always lead to new innovative thinking. To tackle this problem, Gaver and his team have come up with a probe pack which they use to probe the minds of potential users and gain an insight into how they use their homes:

Domestic Probes:

Real interesting stuff. One of the products produced as a result of this type of requirements elucidation is the drift table

To find out more about Ludic Technologies visit the Equator project website.

LISP and The Unified Theory of Everything

2005-11-28T18:54:00.000-08:00

Spent today at XPday London 2005. It was a Great day. I Listened to Tim Lister give the keynote speech, an inspirational speaker. I'll never think of dead fish in the same way again, but that’s food (pardon the pun) for another blog.

It is 3am in the morning and my mind is spinning and making connections. Perhaps spending the day with "thought" oriented people has got something to do with it. Anyway I can't sleep until I get my latest connection down on paper (or should I say down on disk).

The serverside "Beyond Java" discussion has really got me looking deeply into alternatives. One of the posts on the serverside had a reference to a modern online book about LISP and Agile development. IIn the book LISP is touted as "the programming language to create programming languages" or something like that. The claim expressed is that LISP gives programmers the ability to define their own language and thus solve their own specific problems in an optimal way.

Anyone that as used EJB, has come across the situation where you’re chosen toolset is getting in the way of you solving your problem. For a simple problem it's over kill and for a real complex problem it just isn't flexible enough. In short all the pre-defined rules which are supposedly there to "help you" are just getting in the way.

Well with LISP, there are minimal rules. LISP breaks programming down to its basic constituents: lists and symbols. If you think about it, all programs (and programming languages) are made up of lists of things. An expression is a list containing various symbols. Literals are just symbols, so are operators, and methods. Symbols themselves are just lists of other symbols. LISP calls the most fundamental symbol an Atom, so '1' or 'A' or ‘+’ is an atom.

Right so the connection. Anyone with a passing knowledge of Physics as heard about the quest for the Unified Theory of Everything. Physicists have managed to explain all natural phenomena by relating them to 4 fundamental forces (Gravity, weak force, strong force and the electromagnetic force, off the top of my head). So why 4 forces why not 40 or 4000?

It is an act of faith in the physics community that there must be a single law out there, a common thread that unifies all four forces - The Unified Theory of Everything. The thought process behind this faith goes like this. If we can identify the common thread, the single underlying common principle, then our understanding of how the universe works at a fundamental level would be much improved.

Anyway Physicists have had some success. By colliding particles with each other at ever increasing energies they have found new fundamental elements of matter. They haven't found the 'atom' yet but they are learning more about the basic building blocks that make up the universe. This research has lead to String theory. Strings are so small that they are virtually impossible to identify empirically, but theoretical physicists have good reason to believe that they exist. So as currently stands Strings are the common glue that holds together everything in the universe, the 'atoms'. So by understanding Strings Physicists hope to discover the Theory of everything. From all accounts they are well on their way.

I'm on a similar quest. I'm looking into the fundamentals of programming. I have a theory. My theory is that at its heart, programming is an intellectual exercise. It is about the ability of human beings to master complexity and chaos through abstraction. As programmers we deal with abstractions all the time. Programming languages are abstractions, libraries are abstractions, objects are abstractions etc. So what are the fundamental forces? What are the fundamental particles? What are the 'Strings' of computer science? I think that LISP may hold the key to these questions. By striping programming down to its fundamental parts LISP allows you to explore the cognitive issues involved in identifying optimal abstractions to complex problems. So in a sense LISP is the String Theory of computer science - the exploration of the fundamental building blocks of the software universe.

To me it is no surprise that LISP derived languages like Smalltalk are still the crucible for innovation in the Software Industry, despite not being at the top of the hit parade.

Connection made and stored - I guess its back to bed.

REST Revisted

2005-11-27T00:13:00.000-08:00

I never did finish off blogging about REST. Well, where I got to in my understanding is that REST is all about state transfer. Moving and changing state between distributed application components in an asynchronous way. To support state transfer REST has a small number of fixed verbs: GET, POST, PUT, DELETE etc. Along with defining the type of state in question (MIME), and the location of that state (URI), thats all you need for state transfer.

My interest is object orientated distributed programming. Well with no new verbs, object orientated messaging is out of the question with REST (look back at my post about sending the message "deleteStuff" using HTTP GET). So for OO the only RESTful option is object transfer (code on demand).

This is cool for an application that is basically asynchronous in nature. What happens though when you need synchronised transactions, say for Croquet? I posted this question on the REST discussion group back in October and I'm still waiting for a reply:

http://groups.yahoo.com/group/rest-discuss/message/5347

REST is also about constraint, and the REST constraints just don't allow for messaging. Objects contain state hence RESTful "code-on-demand" is possible, but they also contain behaviour which is selected through messages (verbs). Message sending is not possible with REST. So replicating objects across several clients and coordinating their behaviour using transactions as required in Croquet just isn't possible with REST (IMHO).

It would be interesting to hear from anyone who thinks otherwise.

In agile development there is a saying "The simplest thing that can possibly work, but no simpler". For state transfer of static and semi-static web content then REST is that simplest thing. For interactive and dynamic multi-user transactions, such as in Croquet (or online shopping/auctions?), REST just doesn't cut it IMO. So it turns out that REST (http) is what I've allways intuitively felt it to be, a limited protocol for a limited set of applications.

Back to the future with Croquet

2005-11-16T14:50:00.000-08:00

The Squeak community have been pretty busy of late. It looks like they've re-invented the operating system, distributed computing and the internet all in one go . See the Croquet website

This stuff looks like something out of a sci-fi movie, and yes, it actually works!

I've been airing my views on programming in a discussion thread on theserverside. The thread was started in response to Bruce Tates new book "Beyond Java". A lot of people are looking for a better way to develop software. There is a lot of ignorance out there, especially when it comes to anything other than C/C++/Java/C#. Seeing what has been achieved in Croquet using 1970's ideas, has really got me thinking.

What is it about our society that we only value things that have popular appeal? If something isn't popular then we instantly dismiss it. Very often, whether something is popular or not just comes down to circumstances, marketing and luck. One of the most strongly argued alternatives to Java on theseverside was something called MDA (Model Driven Architecture). It turns out that MDA is just CASE reborn, with the same old code generation that failed in the early 90's. Why would anyone want to try CASE again? Yet there still seems to be an appeal. I guess it must be the bang whiz appeal of flashy graphics.

I'm dead certain that late bound languages like Smalltalk will now take off on the back of Croquet, but I can't help but wonder where we'd be today if we spent the last 20 years building on sound engineering principles.

The term "Software Engineering" has gone out of vogue in recent years. Many in the Agile community have turned away from it. I find this a bit of a shame. Engineering is a practical profession, solving real world problems in a practical way. I don't see why engineering should not be viewed as creative. Instead titles like Analyst, Architect and Coach have become popular.

Hopefully Croquet will make an indepth apreciation of technology and the title "Engineer" popular again. I remember reading BYTE magazine in the 90's. Technology would be put through its paces. It was really interesting. SIGs and ACM publications were available too. Today this type of indepth analysis just isn't readily available. Instead technology has become like pop music - you use whatever is currently top of the pops.

We need a new BYTE. Hopefully the achievements of Croquet will make Software Engineering popular again. Developers may start to take more of an interest in how the Software technology they use actually works. Relying less on vendors and more on their own informed judgement.

It looks like interesting times ahead.

A TDD Episode

2005-10-21T03:02:00.000-07:00

I'm at the tail end of a hairy refactor using JFreeChart. My client has assets which they need to manage the quality of. The application uses a JFreeChart to show a Colour Coded Quality (CCQ) representation of the asset. The Y axis identifies four quality measures and the X axis is the longitudinal position along the asset. A colour coded item at the X/Y vertex indicates the asset quality at that location and for that quality measure. (Sorry about the terse description, but I'm trying my best not to give away the identity of my client).

Anyway... There is an existing implementation, but the data generation, plotting, and rendering are all terribly intermingled. The rest of the app has been rewritten to get away from an abstract data model (don't ask, it ran like a dog), but the new charting was still yet to be done. So we needed the old chart, but we didn't want the old data generation bit.

How to proceed? Well I started with a standard JFreeChart Plot that sort of did what we needed, a ScatterPlot - A GREEN bar. So what to do next? Well the guts of the dataset code could be clearly identified scattered around the old code, so I copied and pasted that code from the old to the new. A few changes, and the creation of some canned data and... GREEN bar ( a running scatter plot using a CCQ dataset). The CCQ Chart shows rectangular data points of varying colours (DataItems) depending on the quality. For example a "compliant" dataItem is green and a "failed" item is red etc. The old renderer did this so we copied and pasted the old rendering code - GREEN bar (a running scatter plot with CCQ data and CCQ colour rendering). Along the way we identified lots of small refactors that needed to be done, but we just to a note of them. After all we didn't have a true green bar as our plot wasn't exactly the same as the original. We did the same process with the "Plot" and... Again with a bit of work GREEN bar.

Now we had the three main components needed for a chart: a custom dataset, renderer and plot. Our chart factory was still the original Scatter Plot, but everything else had been migrated to custom CCQ charting code. Yet our plot didn't look like the original. We were missing the labels on the Y axis and couldn't work out why.

After a while head scratching we decided to plug our new dataset, renderer and plot into the old code. We then refactored the old "factory" until it was all in one place and looked similar to ours. After each small change we would run the code and check that everything was still GREEN. We finally got down to the final difference. The refactored old code didn't set the chart orientation, relying on the default our new factory did. So we decided to add it this line of code to the old code and BANG, RED bar. It turns out that all we needed to do was remove this line from our new code - still not sure why though, possibly a bug in the JFreeChart library.

So time for a mini retrospective. What where the lessons learned:

You don't need JUNIT to do TDD. Often if the code compiles and runs then that is a good indicator of a GREEN bar. You may only have partial functionality, but if your code does what you expect it to do, then that's a GREEN bar.
Refactoring to patterns really works. It allowed us to make some real big refactors in one step. We identified our target design by looking at the JFreeChart ScatterPlot code.
When refactoring on the grand scale, getting the overall structure right and getting back to a green bar as soon as possible is the main goal. Small local refactors can happen later. By keeping a list of refactors you are able to choose the order in which to do them.
always "listen" to the code. Code smells will tell you what to refactor. Commenting out declarations and seeing the resultant errors in your IDE also tells you a lot. A refactor with a lot of dependencies isn't a good candidate to do first, because it could leave you on a RED bar for a long time. Do something easier first, and get back to GREEN .
If you find yourself on a RED bar for a long time, then undo the change, get back GREEN, and make a series of smaller changes. This is how we identified the orientation bug. If an ambitious refactor is failing you always have the option to go back and take smaller steps.
Small refactors are better. In hindsight, after identifying our target design pattern, we should have refactored the old code to the new design in small steps, rather than copy and paste. We would probably have maintained a better rhythm and learned more about the old code, whilst spending shorter intervals on RED.

How ReSTful are you?

2005-10-20T23:39:00.000-07:00

I've joined the REST discussion group on yahoo groups. And a very knowlegeable and friendly bunch they are too.

After following the group discussions I'm beginning to realise that their is a bit more to this REST stuff then I first thought.

For example, one would assume that Hessian Webservices is pretty ReSTful (obeys the REST constraints). After all what could be wrong with:

http://myserver/myServiceInterface?method=doSomething&param1=value1


Where 'myServiceInterface' is a remote web service and 'doSomething' is the method on the remote interface you wish to invoke.

Well the uniform 'HTTP' connector has its own generic commands: GET, PUT, DELETE, POST etc. So what happens when your client sends:

http://myserver/myServiceInterface?method=deleteStuff

Using a GET?

REST: A not so Uniform Connector Example

2005-10-16T00:39:00.000-07:00

Still buzzing with the possibilities of REST. One of the things I don't like about http is that it has a tendency to reduce everything down to the lowest common dominator "Strings". So all your parameters are strings and often your data is one long string in the form of XML. This is as a result of the REST constraint for a uniform connector across the entire web. But the world isn't uniform, it contains different realms each realm with its' unique characteristics. Isn't it suffcient that your connector is uniform within its given realm?

For example say I want to get access to a resource on a remote server running Squeak. Why not create a squeak specific connector by sublcassing:

SqueakConnector subClass HttpConnector
" A connector that understands the Squeak object representation"

formUrl: squeakServer object: anObject: message: aMessage params: parameters
"Answers a URL object"

GET: url
"Answers the squeak object at url"

A Squeak client would look like:

con := SqueakConnector new.
url := con formUrl: #myServer object: #UserDB message: #getUser params: (Dictionary new at: #user put: 'fred').
fred := con GET: url.

The actual URL passed to "HttpConnector>>GET" would be:

"http://SqueakUniverse/myServer/UserDB/getUser?user=fred"

As long as all servers in the "SqueakUniverse" agree on their object representation and use an agreed URL format then this should work. And my client code doesn't need to worry about dealing with the actual object representation on the wire.

I'm sure this is still RESTfull, honouring the concept of a uniform connector, but constraining its scope to a specified namespace. I need to see if this type of stuff is allready being done.

REST and OO Abstraction

2005-10-15T13:58:00.000-07:00

I'm still getting over just how simple REST is. In my first post on REST I was wondering where a protocol like IIOP fits in. Mark's comment kindly explained, that as far a REST is concerned it doesn't. So I've been thinking how does abstraction fit into REST. In REST a datum is data that is exposed by a component. Which makes sense, since data that you want to communicate to another component cannot be hidden. So the datums exposed by my object are resources that I can lookup:

// client side lookup
datum1 = connector.GET(
"protocol://myServer/myServiceObject/Datum1");

OK this is pretty fine grained, what if I want to get a lot of data at once. Well I can think of two options. Option 1: Expose all of the data in my object and send it through the connector to my client. This is what IIOP does through marshalling. What IIOP goes on to do is to un-marshall my data back into an object at the client end which breaks the REST constraints on a connector.

So without unmarshalling we have:

// client side lookup for a data structure
struturedDatumXML = connector.GET( "protocol://myServer/myServiceObject/ObjectDataAsXML");

// data structure access
structuredDatum = new XMLObject(struturedDatumXML)
datam1 = struturedDatum.getElement("datum1").getValue();

But we've broken encapsulation, so the messaging is no longer Object Orientated. Going back to IIOP my structuredDatum would be a data transfer object (DTO), with a set of getters for each data element. But a DTO is really just a data struture also, so IIOP doesn't help. So how can we do this in REST, and not break encapsulation. Option 2: How about mobile objects:

// client side lookup for a real Object
objectDatumSerialised = connector.GET( "protocol://myServer/myServiceObject/Serialise);

// client side use of object
objectDatum = ObjectLoader.load(objectDatumSerialised);
datum1 = objectDatum.getDatum1();

Treating an object as a resource should be possible with VM based languages.

So after my fears about abstraction and encapsulation, it turns out that the REST constraints do not preclude abstraction at all. I'm new to this REST stuff, so the above could be very wrong and all comments are welcomed. In particluar I'm interested in finding out more about "self describing data" as it sounds like a mechanism that will help abstraction.

So it looks like I don't need to loose Object Orientated abstraction to use REST. Until someone explains otherwise.

REST is so Simple

2005-10-14T01:57:00.000-07:00

In my last post I was struggling to understand REST . After looking at this example it all made sense. Almost too simple to be true. After reading the dissertation , I was expecting REST to be a lot more complex than it is. In practice the REST style (set of architectual constraints) is simplicity itself.

If you intend to read the dissertation, make sure and read chapter 1 before reading chapter 5. Chapter 1 gives a precise definition of terms, which you'll need. Or if you're like me, just jump straight to the example. Don't be put off by its brevity; connectors, components and datums as defined by REST are all there.

Will REST win out over SOAP? Well I for one won't be using SOAP/WSDL/UDDI etc. Why would anyone want to use that stuff given the choice?

REST to the Rescue?

2005-10-13T14:04:00.000-07:00

I've heard a lot of good things about REST (Representational State Transfer). Apparently REST represents a distributed computing architectural pattern that brings simplicity and consistency to the WEB. Well at least that is what I have been lead to believe. Anyway I've finally found a description of REST, that I can understand.

After reading through the description (and it is quite lengthy) I was somewhat disappointed. I'm not sure why, but some how I was expecting more. I come from a telecoms background and I am pretty familiar with network protocols and communications. I remember reading the GSM protocols for the first time, I was immediately impressed. GSM is a very complex system with a number of orthogonal protocols that work together to address the various issues associated with a single application, mobile voice and data services. After reading the REST description, I was left thinking what is this architecture meant to do? What problem is it meant to solve?

This seems to be the key issue. With the exception of hyper-links, there doesn't appear to be a generic web application, instead their is a number of disparate applications that happen to share the HTTP protocol (out of convenience) and URIs as a common addressing mechanism. REST some how tries to come up with an architectural pattern that addresses all these disparate needs, and in the end becomes so unspecific it is almost meaningless (IMHO).

The only benefit I can see, is that it dissuades extensions to the web that are not consistent with good network communications practice. It appears to be an attempt to limit some of the network protocol abuses that have characterised the web from its inception.

What is missing is specific protocols for specific applications. For example, I once hoped that IIOP would become the established protocol for client/server applications over the Web. In my opinion HTTP is just inappropriate for such applications and if the W3C promoted an alternative I'm sure firewalls would be opened up (in the same way that ftp is accepted as the protocol for file transfer, telnet for terminals, POP3/STMP for mail etc).

In the article it talks about mobile code, but again nothing specific. I may be missing the point here, In which case can some one please enlighten me. I was hoping that REST would be the antidote to SOAP,UDDI,SOA etc Please tell me that my hope isn't misplaced.

The Backlash Against Open Source has Started

2005-10-12T07:31:00.000-07:00

I've recently been looking into Open Source Software. Whilst searching for blog entries on the subject, I came across this post.

It's worth a read. The basic gist of the argument made against open source is that open source developers are being exploited by unscrupulous Software Vendors. To stop 'exploitation' the author suggests that the government should step in and 'regulate'.

It may come as no surprise that the author is a Microsoft supporter.

If anyone out there comes across other interesting blogs/articles on Open Source please let me know.

Is .NET (Mono) The Open Source Platform of the Future?

2005-10-03T15:33:00.000-07:00

Been looking into Mono, The Open Source .NET clone. In an attempt to win people over from Java it appears that Microsoft has gone to some lengths to ensure that .NET is percieved as open.

In fact, my understanding is the the .NET VM (Common Language Runtime or CLR in Micorsoft talk) specification has been submitted to the ECMA, an international standards body.

I guess that the people at Microsoft thought that as long as they could control the .NET implementation, then .NET would be proprietary. But with an open standard anyone can create an implementation.

The Mono implementation supports Windows, Linux and OS X on the Mac. In addition, Mono has ported a broad set of languages to the .NET platform, including Ruby.NET, #Smalltalk and even Java.

I first thought that this common language VM thing was just a marketing ploy. But if you think about it, it is a great idea. What normally locks developers into a given platform is the supporting tools. But with a VM your code will run on any platform that the VM has been ported to. Thanks to Mono, for .NET that means all major platforms. The secound barrier to developers changing technology is library support. If the only good user interface library for Windows is written in C#, then I guess your stuck using C#, but with .NET/Mono you could choose to program in say Ruby.NET and still use your C# libraries.

So you program in Java, but there is a great library in GtK#, no problem, just compile your GtK# libraries into a Java jar (Java stubs) and link at run-time.

I'm not sure just how good Mono is, but I do know that it is backed by Novell, and with time it can only get better. From the look of things on the Mono website, it is more than ready for production use. The other concern of mine is that I'm not sure just how well the CLR supports languages other than C#. In particular I'm not sure how much support is present for dynamic languages like Smalltalk and Ruby. If the CLR byte code does support both dynamic and static languages fully or is extended to do so, then .NET has got to be the ideal open source platform.

Think about it. You decide that Ruby is more productive than C#, well just use Ruby.NET. You've got loads of Java libraries you want to use, well if you've got the source, just re-compile for .NET or if not dynamically translate the jars to .NET at runtime.

I don't think the Java/J2EE world has an answer to this kind of flexibility. The EJB monolithic container model has failed, and the light wieght, mix and match services a la carte model is available on .NET as Spring.NET and Hibernate.NET.

I'm keeping an eye on Mono. If it delivers, we as developers will be free to use what ever platform, language or libraries we choose. And with .NET locked into the ECMA standards process, I don't think Microsoft can do anything about it.

Microsoft setting the world free. Isn't that a thought!

Open Source - Who's doing it and why?

2005-10-03T03:19:00.000-07:00

In my last post I discussed the effects of open source software on the Java community. In my opinion, Java was a reaction to a perception by many in the software industry that unless something was done that Microsoft would end up owning both the desktop and the server. If this was to happen, then in time Microsoft would be sure to use its dominant position to shut out other Server Software Vendors. After all, this is exactly what they did with Lotus, WordPerfect and others on the desktop.

As a consequence Sun, Oracle, IBM et al got together and agreed that the Java VM would be "the common means of passage" on the server. Cross-platform, Machine independent, OS-neutral and with open APIs, Java would fend off Microsofts' attempts to own the Server.

Interestingly Java has become a run away train, no longer under the control of the vendors. The developer community have taken the open Java APIs and crafted tools and APIs of their own, all open source. So who owns Java now? Well it appears to me that the open source community are in the driving seat.

So who are the open source community? Well things have changed a bit from the days of Richard Stallman. It appears that many open source projects have big backers. The Mono project, a .NET clone is backed by Novell, and IBM backs Eclipse (the leading Java IDE). Linux has become an open source phenomena in it's own right, with backers such as Sun, Oracle and most of the big players. The only large player not to be some how involved in open source is Microsoft.

Open source has become a viable business model for many. Look at Skype which was recently sold for billions. A less dramatic open-source business model is to offer consulting and bespoke variants on the back of an open source product. It seems that different people are involved in open source for different reasons.

I have always thought that their should be a "common means of interaction" in software, much like language or law in society. This common base should be freely available to everyone and owned by no one. This seems to be exactly what we're now getting in Software.

But why are the vendors backing it? One argument is that producing software open source is cheaper. True, but you can't sell it. I've read an article that suggest that open source is being used as a means of wrestling market share from the incumbent monopoly (Microsoft). It is difficult even for a monopoly to compete with something that is free. Once the free software becomes the de-facto standard (70-80% market share), a new market is available where the backers of the open source product can sell consultancy services and add-ons. I believe this is already happening with Linux.

For example, I wonder how many Oracle licenses have been sold under Linux. Now without the success of Linux, what percentage of those licenses would have gone to SQL Server under Windows? Most right.

If this strategy is true, then this sounds like a dangerous game for the vendors, as they are sure to loose control in the same way that they have lost control of Java. And in a world where customers feel empowered to pick and choose their own software without vendor endorsement, innovation and new ideas could quickly leave the vendors flat footed. Look at the emergence of Ruby on Rails.

Anyway I'm going to get myself a good book on open source. I think it's time I took the open source movement a lot more seriously.

Java for Sale

2005-10-01T09:56:00.000-07:00

I was an early adopter of Java. I was easily persuaded by the arguments in favor. A few years before Java came out, I had dabbled with Smalltalk, so I knew the advantages of a VM. Besides things couldn't go on the way they were with C++ and memory leaks.

Back then Java still had many detractors. Like with Smalltalk, performance was the main criticism. Yet I knew that micro-processors would get faster over time and Java was fast enough for most applications even back then (desktop GUIs were sluggish though). I knew Java wasn't as elegant as Smalltalk and the use of a VM wasn't anything new, but Java was type safe, C-like in syntax, and reasonably performant. For the majority of C++ developers trying to produce server side software, I could see that Java and J2EE would be a great leap forward.

It took me about 2 years to convince my company to use Java. By this time Java had gained quite a momentum in the industry. J2EE Application server implementations were beginning to emerge, and consultancies like Valtech where championing Java as the technology of the future. In fact quite a few big name companies had thrown their weight behind Java. By this time Sun had repositioned themselves as a software company and were practically betting the business on Java. Oracle had jumped on the bandwagon, and as well as creating their own J2EE App. Server, had written Java into the core of their database product. Initially IBM were on the fence, backing both Smalltalk and Java as the OO champions that would steal developer mind-share from Microsoft. Over time though IBM reduced support for Smalltalk and became massive backers of Java (I think even bigger backers than Sun).

I could see the power behind the Java promise of "Write once, run anywhere" especially in light of the threat from Microsoft. Without Java, we could all end up Microsoft developers, with anti-Microsoft vendors forced out. Microsoft had shown their intent for world domination by refusing to "play fair" within the OMG. It was all out warfare.

The anti-Microsoft camp was throwing Millions into Java, downloadable free over the internet. The thought crossed my mind more than once, "How did these companies expect to make money?" If Java software was available free, then developers would try it? Yeah OK. So if developers were using Java then companies would need to support it? True, but what could you sell to companies?

The first commercial Java tool that I used was WebLogic (no not BEA WebLogic, Just WebLogic as it was back then). This was a relatively light weight App. Server written totally in Java. Easy to use, and configured through a single property file. It worked a treat. A developer license of £2K per developer and a run-time license of £11K per server seemed a bit steep, but was reasonable given the gains in developer productivity.

But none of this money was going to Sun. How was Sun going to make money out of Java? I'm not sure if the people at Sun knew, but if world domination was the name of the game, it would be better if the glue pulling everything together on the server was the Java VM rather than Microsoft Windows OS.

The first big non-commercial Java tool I used was the Apache Web Server with built in J2EE Servlet support in the form of Tomcat. The Tomcat servlet engine perfomed much of the server side resource management that full blown App. Servers did, and it wasn't long before some began to recognise that for most applications, the full J2EE stack (Servlets, Stateless/Stateful/Entity Beans, JMS, JNDI, JTA , etc.) just wasn't needed. Besides if you really did need a full J2EE App. Server you could download one for free, JBoss.

By this time the marketing machine behind Java was in full swing. The App. Server market had matured with IBM's WebSphere and BEA WebLogic as the market leaders (Sun missing out again). These new App. Servers weren't simple and clean like the original WebLogic. Instead, they were cumbersome and clunky. Awkward web based App. Server administration was in vogue, and complex XML application configuration could take weeks. Feature bloat had set in, and they where sold as all things to all men. The compexity of these App. Servers was a real problem. Class loader hell, was a common experience for many J2EE developers, and gaining mastery over your App. Server took time and dedication.

Microsoft had woken-up to the marketing power of the Application Server. Middle-ware was the new industry buzz. After being forced to give up on "Microsoft Java", Microsoft produced their own "clean-room" Java clone, C#. They then went on to market their new middle-ware vision. In addition to "Write-once run anywhere (Microsoft chooses)" they added "Write in any language you like (if we choose to support it)" and "Expose your business services over the web (to anyone who uses are protocols)". The .NET era had arrived, it would take about three years before working .NET code would ship, but in marketing terms .NET was real and the industry was holding its' breath.

Sun had put in place the JCP process as a "community" lead way to create and improve Java standards. The problem was that the JCP was a community of Java Vendors not Java Users. Sun has allways flirted with the open-source community as a way of gaining credibility with developers. The fact that you could download the Java JDK source code for free, made Java appear almost open-source, yet there was a growing group of vendors who had bet a lot of money on Java and were looking for a return on investment.

Every .NET announcement would result in a corresponding JSR being raised to provide the same functionality in Java. The Vendors were playing the "my App. Server has more features than yours" game with Microsoft ane each other. Whilst this was going on, the developer community were realising that much of the stuff coming out of the JSRs didn't actually work that well (EJB 2.1 Entity Beans comes to mind), and they were busy producing better open-source solutions themselves. Eventually the gap between what worked and what the JSRs had specified became untenable.

Spring and Hibernate showed what could be done when you actually tried to solve problems rather than sell product. The cat was out the bag and all the JCP could do was try and standardise what the developer community had allready chosen for themselves. I would argue that this is exactly what the JCP should have been doing all along.

They say you can't serve two masters. The JCP process can't be in the interests of both Vendors and Developers at the same time. The developer community have taken matters into their own hands and are busy shaping their own future. Microsoft has gone very quite with .NET recently. The open-source community has targeted .NET too with Mono. The Java Vendor community are still trying to pretend that it is business as usual with JSF (JSR127) and EJB3 (JSR220). The JSF technology is practically obsolete before the first decent vendor tool has been shipped. Everyone knows that the future of web development lies with Continuations and Ajax, both missing from JSF, but available open-source (Squeak Smalltalk and Seaside). EJB3 is available open-source too, as Spring and Hibernate. Why wait to pay for a buggy Spring/Hibernate implementation from BEA?

So my take is that Ruby will carry on picking up momentum on the back of Rails, and who knows even Smalltalk may see a resurgence in popularity with Seaside (I hope). As for Java, well its' still for Sale, but I think there will be less takers in the future.

The Economy: China to the rescue?

2005-09-25T16:19:00.000-07:00

Just watched Panorama ( a BBC documentary), on Gordon Browns 'Economic Miracle'. Apparently what has sustained the longest period of economic growth in recent history has been consumer and public spending. Even with spending, inflation has remained low. Low inflation has been as a consequence of cheap imports form the far east and China. This as allowed us to avoid the boom and bust economic cycle of the past (high spending-> high inflation -> high interest rates -> economic slow down).

The growth of imports as been at the cost of local manufacturing. So we no longer build stuff, but stay afloat by selling things to each other built cheaply else where. And of course there's the growing 'service sector' what ever that is (retail parks, DIY super stores, etc).

I'm no economist, but I know that most people feel pretty insecure at work. Most people are working harder for less. Job stability is a thing of the past, and most people have come to accept short term contracts as part of life. The only thing that seems to produce 'a feel good factor' is house prices - and who knows when that will suddenly come to an end.

The scariest thing that came out of the programme for me, was that once UK manufacturers had moved their production to China, the benefits to the UK economy (lower retail prices) will be realised the once. Once things have settled down, low prices and consumer spending, can no longer be relied upon to sustain future growth. Worst still, without a manufacturing base of our own, we will become sensitive to price inflation in China. Should the Chinese decide to pay themselves more, then that will be reflected in higher prices in the UK.

So after years of protectionist EU trade policies on food. We will suddenly find ourselves dependent on others when it comes to manufactured goods.

What will become of the average man on the street? After all we all can't work in retail parks, and I don't think we're going to loose our appetite for consumer goods any time soon. Maybe the French have the right idea, and it is time to start putting up the barricades!

It really does look like unstable times ahead. Economic power in China, Military power in the US, the Europeans in the middle. Friction between China and the US seems inevitable to me. Some how I feel that the Americans with their vast resources and dynamism, will be able to meet the Chinese challenge. I fear that China's gains will be at the expense of the Europeans, including the UK.

I don't fancy learning Mandarin, so it looks like I need to get myself a Green Card quick!

Software Vision - Creating the Backlog

2005-09-20T11:24:00.000-07:00

On the back of my last post. I've come up with an idea. My criteria for envisioning new systems:

Each release should be take no longer than 3 months and deliver usable value
The complete vision may take a number of releases to realise
The system/product definition (backlog) should be owned by customer/marketing
Customer/marketing should decide how much 'traction' is needed between releases

The last point may need some explaining. By traction, I mean the degree of organisational change required to support a release. In circumstances where organisational change is difficult, it may make sense to use a 'low gear' introducing small organisation changes between each release, providing more 'traction', and increasing the possibility that the organisational change will stick. Conversely if the going is good, then larger changes may be possible, allowing the end vision to be realised sooner.

The point here is that software vision is a skill, and the people responsible need training. So my bright idea is training for customers/marketing people on how to create a product backlog.

Agile development starts with the backlog. But what happens before that? Here is my suggestion:

A new product idea (instigated by anyone)
If idea meets some minimal criteria, it enters the project funnel
Organisation use some criteria to decide which projects to fund and in what order
Funded projects acquire a 'customer team'
Customer team go through some training on creating 'product backlog'
Customer team create first pass backlog
Development team work with customer team to refine and realise backlog

I'm going to give some thought to the training and assistance 'product' teams need to envision a good product backlog (product definition). I will use the term 'product team' instead of customer team as it is consitent with 'Product backlog' and SCRUM.

Hopefully I can come up with some good guidelines.

Anyway watch this space.

Software Vision

2005-09-19T12:02:00.000-07:00

Alan Cooper and Kent Beck debate XP vs Interaction Design. Alan Coopers' book The Inmates are Running The Asylum describes Interaction design.

Not sure what 'interaction Design' is, but from the article it appears to be some type of business analysis that occurs prior to software development. The assumption is that users need 'help' in deciding what it is they want built. Rather than just automating what is, we should be designing something better. Sounds similar to the Business Process Re-engineering ideas of the late Nineties.

In my post on software as art, I point out that there isn't much guidance available on how best to envision new software systems. I'm not sure whether having a middle man between customers and developers is the right idea though.

It is all well and good, if your Interaction Designers are good at what they do and add value, but how do you test this? Also how do you keep the customer accountable for how he chooses to spends his money?

All sounds a bit naive to me.

Envisioning should be a customer/marketing responsibility, with developer input on feasibility and cost.

I'm with Kent Beck on this one.

The Verdict - NLP is Mumbo Jumbo

2005-08-23T08:54:00.000-07:00

Past experience has taught me humility. Hence my criticism of NLP has been somewhat guarded.

Having looked up NLP on Wikipedia, I am pretty convinced that NLP is pseudo science. If the lives of the inventors of NLP are anything to go by, then NLP is definately dubious to say the least. On wikipeadia, there are links to several critiques of which this one is typical.

What does NLP say about our society? Why is everyone looking for a quick route to "success"

I'm of Jamaican decent. In Jamaica, especially in rural areas, old west african values still hold. People are a lot more relaxed about life, a lot more social, and a lot happier. In the town however, (kingston) european values tend to dominate. In a small island, the two cultures sit uneasily together.

When visiting Jamaica, it is clear to me what we in the west have lost. It is also clear how global economics is forcing much of the world down the same path.

People need to ask who is benefiting from this trend? Certainly not the majority of the worlds population. Where will it end? 1 billion chinese are about to join the rat race in earnest - can the planet support this?

Frankly I find it worrying.

Materialism, Spirituality and NLP

2005-08-23T05:33:00.000-07:00

Been reading more about NLP. From the moment I was introduced to NLP, I've been a bit uncomfortable with it. In my last post on NLP, I questioned whether NLP could help if your desired outcomes where themselves spiritually unforfilling. Well, the answer is sort of. At least from what I've read. In choosing outcomes, NLP suggest that you identify outcomes that 'suite you'. Outcomes that are congruent with all aspects of you. NLP goes on to say that change should be supported by your subconcious.

NLP also describes a concept known as modeling. This is where the behaviour of "successful people" is analysed and copied. In this way NLP believes that success can be taught.

Unfortunately in my reading thus far NLP has not defined what it means by success. From what I've read, the implication is material success.

So if material gain, is the yard stick for success, where does the soul fit in? It seems to me that this focus on materialism turns sprituality on its' head. Rather than seeking contentment, peace and happiness from within, NLP seems to be promoting the search for happiness through external things.

I think this is a sign of our modern times. Production and consumption, is our new God. As human beings we seem to have lost the connection with our own souls, with nature and with God.

NLP seems to be trying to harness the resources of the soul in the service of material gain. If my interpretation is correct, then us in the west are truly lost. Here is a quote:"... In this spiritual confusion, many cults, sects and - issuing also emerge. Both religion and philosophy become materialistic and politicized...". This is taken from Brahma Kumaris

Life is speeding up, and increasingly more materlisitic. In the west, we are less communal, more individualistic and increasingly isolated as individuals. In this all consuming rush for wealth, our humanity itself seems to be the victim.

Creativity, Simplicity and Art

2005-08-19T02:54:00.000-07:00

I've been thinking some more about "software as a creative process'. Good software can be a thing of beauty. Simple idioms and patterns repeated to produce something that is both complex, yet simple. Much like the double helix structure of DNA or fractal patterns in maths.

Beauty in software has two main manifestations. Firstly in its' conceptualisation. Knowing what problem to solve, and what the solution should be, is a creative process. One instantly recognises a good solution to a problem. A good solution is often simple, and a good fit for it's intended purpose. When going through open source projects on sourceforge, good project ideas immediately jump out at you. A project that meets a real need, and is simple.

The second manifestation, I think, is in the structure of the solution itself. Object orientated languages offer great scope for efficient, elegant solutions, where functionality is implemented once and once only. Dynamic OO languages like Smalltalk, offer even further scope for elegance and simplicity. A dynamic language can greatly improve productivity, allowing time for trying things out. Learning through discovery, leads to increased beauty.

Unfortunately, the beauty of a software concept, although visible to the business, is often overlooked. Very few business leaders spend time agonising over whether a proposed software concept is beautiful. Software just isn't thought of that way. More important to most business leaders is the projected return on investment. I've never calculated ROI, but I'm sure that it is a difficult thing to predict. Also I would hazard a guess that projects that actually do provide a good ROI are both beautiful and simple in concept.

Worst news is that the beauty of the final implementation is not visible to the business at all. The first the business becomes aware that a software solution may be less than beautiful, is when bugs begin to emerge, or when maintenance is more difficult and expensive than anticipated. The developers know when a software solution is ugly, but unfortunately, the business hold the purse strings and make the final decision.

Implementing beautiful software is a creative skill that can be taught. Much like painting or playing a musical instrument. The art of software development is well established, and there is a gamut of practices and disciplines for developers to draw upon. Agile development recognises the creative nature of software development, and promotes practices that support creativity. In contrast software conceptualisation, seems to be less well understood. The nearest I've seen to a disciplined approach to deciding what software to build is the approach outlined by SCRUM, namely keep it small and keep it simple. Beyond this there doesn't seem to be much guidance around unfortunately.

So where is the software industry today?

Well we've got managers who would like to think of software development as a defined process. Gant charts and LOC estimates, leaving little scope for discovery and creativity.
We've got developers, who tend to be more interested in technology, then delivering business value.
Where beauty is visible, in conceptualisation, the art is not well understood.
Where the art is well developed, in software implementation, the resultant beauty is not visible.
The business care little for conceptual beauty, as they do not appreciate its' importance. Developers can avoid discussing implementation beauty honestly with business people as the implementation is no visible.

It would be interesting to see what business leaders from creative industries like fashion and music would make of the software industry. I'm sure they could teach us a lot.

Coaching and NLP

2005-08-14T04:16:00.000-07:00

I've not blogged about my agile coaching role for a while. To be honest, I'm a bit disheartened with coaching. Coaching as I understand it places the emphasis on the person being coached to create change. Whilst on the surface this makes sense, it does assume that the person being coached has positive goals and is motivated to achieve them. If this is true, then the coach is a mere assistant showing the way.

Whilst many sports men and women, do have elevated ideals, I don't think the same can be said for the workplace. My experience of the workplace is that of extreme politics. By politics, I mean numerous agendas all vying for supremacy. In my experience, very few of these agendas can be described as noble, aimed at making the world a better place. In my opinion , these competing agendas are responsible for making organisations sub-optimal, and less than ideal places to work and thrive.

Some organisations do manage to maintain lofty ideals, with everyone working towards a common goal. Organisations such as Universities for instance, where an ethos of egality and openness is well established. In such organisations, motives tend to be somewhat different than in the workplace. The pursuit of personal financial security (money), is largely replaced by the desire to gain the respect of your peers.

In the workplace the need to be held in high regard by others can exist too. The success of Japanese companies like Toyota for instance, can largely be put down to the well established clan system that has existed in Japan for many centuries. By fostering clan like loyalty, Japanese companies have managed to achieve a high degree of optimalisim.

In a last ditched attempt to see if coaching can really address the problems in the modern work environment, I've started looking into NLP (Neuro Linguistic Programming). NLP is a psychological science touted by many in the coaching profession as the theoretical under pinning for what they do.

NLP is an interesting mix of eastern leaning philosophy, spiritual awareness, and modern western psychology. NLP proclaims that we can better achieve our desired outcomes, by re-programming the way we think, speak and behave, and in so doing become successful. In NLP, the role of the coach is to help people 're-program' so that they can succeed.

As someone who is well versed in eastern Buddhist philosophy, and who believes in the power of meditation (directed thought), a potential flaw in NLP jumped out at me straight away. NLP starts with the desired outcomes of the practitioner, but very often, it is these desires, that are at the root of the problem.

To explain this last statement fully, would take more time than is warranted here. But in short, it is what we desire (our intended outcome), that is often what leads to our unhappiness in the first place. For example, many of those who desire 'financial security' still find themselves feeling 'insecure' and desperately unhappy, even when they have acquired more than adequate wealth.

People caught like this do not fully understand their own motives. They are consumed by their surface desires, unable to see what lays underneath. Often their desires stem from uncomfortable human emotions, such as fear, jealousy, low self esteem etc. Free of the need for a quick fix, they are more able to see what ever it is that will make them truely happy. I describe this type of awareness as wisdom.

This is why, I feel that what is needed in the workplace is leaders. People that can lead by example, act as role models and inspire those around them to elevate their thinking and gain wisdom. Such people don't merely facilitate, but set the direction and promote values that others can follow. In my coaching, I make a conscious effort to lead. This can manifest itself in several ways. One important way is establishing congruence between my actions and my words. By this I mean 'walking the talk', practicing what I preach. Through this I manage to gain trust, a commodity that is in short supply in the workplace. Next, I try to limit my tendency to judge others, and to become over critical. This is something that I really struggle with, but when I do achieve it, I find that it increases my ability to influence others. People tend to be more open to you when they feel safe from personal attack. Finally, when necessary, I'm brutally honest, and unequivocally is saying what I feel is needed. After a while, when people feel safe and that they can trust me, they become open to taking my lead. I find that when I talk, most people are willing to listen, especially when I take the time to listen to them.

Despite my personal achievements in persuasion and leadership, I find myself in scenarios, where a positive coarse of action is being blocked by someone with higher authority. In most cases this person has sanctioned my role, but has chosen not to consult me on a decision that affects my ability to do my job. They are happy to pass on responsibility whilst reserving control for themselves, leaving me powerless.

This in my opinion, is a failure in leadership. Most often triggered by fears. For such people, what is missing is the ability to lead themselves. To conquer their fears and self doubts. Leaders aren't born in my opinion, but made. The concept of leadership is well established, and many cultures have established ways of ensuring that they foster good leaders. With a good leader a coach becomes a useful tool, helping the leader and their team(s) become the best they can be. The sports men and women, who use coaches to achieve their goals are a good example of this. A sports person, who does not pursue excellence, or who is not motivated to do the hard work needed, will not be helped by a coach.

So whilst the role of coach is significant, what I feel the workplace really needs is good leaders. Not being a captain of industry myself, I have limited my ambitions to leading my own life. NLP may help me get to where I'm heading faster, but if I'm heading in the wrong direction does NLP help?

I would love to hear from soneone with real experience with NLP.

NASA finally sees the light

2005-08-03T06:53:00.000-07:00

It looks like common sense has finally prevailed at NASA. You have to give them credit:

Space shuttle replacement

Meanwhile the Russians are having a quiet chuckle, whilst planning their own future:

Russia in Space

In my last entry, I couldn't remember the name the Russians gave their rockets. Well it is Soyuz.

BTW the russian Soyuz rockets date back to the 1960s, and hence are over 40 years old in lineage. Apparently, the Russians have made monumental leaps in rocket technology, eclipsing the early achievements of German pioneers. So perhaps it was abit unfair of me to imply that their current Soyuz rockets are based on German WWII designs - sorry.

The space shuttle in trouble again

2005-08-02T11:44:00.000-07:00

Not wanting to annoy any Americans out there, but the latest episode in the Space shuttle saga, is evidence if we needed it that complexity should be avoided.

Space travel, like software development is inherently risky. So why compound that risk, by devising a space vehicle miles more complex than it needs to be?

The Russians seem able to put people into space, a lot more reliably and at a fraction of the cost. So what is their secret? Well a simple rocket design, inherited from the Germans after World War II, is still used by the Russians today.

This simple design has its' advantages, the dangerous fuel tank, that makes up the bulk of the vehicle, is behind the capsule where the astronauts reside. The cockpit capsule can be ejected from the rest of the vehicle, protecting lives in a catastrophe. In contrast, the space shuttle design, has the astronauts sitting on top of a massive fuel tank and adjacent to two solid rocket boosters, with no escape route.

From a safety view point, the space shuttle design is ludicrous. In terms of complexity, the shuttle is significantly more complex then the rockets the Russians use so successfully.

So why? Oh yes, the space shuttle comes back to earth, this as got to be an advantage. Well no, the Russians manage to travel to space at a fraction of the cost, even though they have no reusable parts.

So what is the real reason? Well if I was to hazard a guess, I would say ego. The same reason why so much software is much more complicated than it needs to be. Having a space vehicle that looks like it came out of a "Buck Rogers" movie is more flattering to the ego, than a plain old rocket as depicted in a bugs bunny cartoon.

I'm sure that national ego, will keep the shuttle program going, when all involved must know that the basic design is fundamentally flawed. I've seen this type of "group think" before, often in companies. Compound a bad decision, by ignoring it, glossing over the facts, and pouring good money after bad. After all we don't want to admit that we got it wrong, do we?

I've been learning Ruby lately, and I've taken a look at Rails. I find Ruby an elegant and productive language (when compared to Java), and for most web apps I've built, I'm sure that Rails would have done the job in a fraction of the time (and cost).

So why are people still building web applications using EJBs, JNDI, XML, JSP, CSS, JavaScript etc, etc? For some, mastering this soup of TLAs is an ends in itself. Being able to do this stuff just makes them feel good. It plays to their ego.

As for me, I get my kicks by knowing that I've produced something useful. Something, that will make someones life easier, better, less stressed etc. So with all this press about the space shuttle, I like to think of the Russians - brushing the dust off their 1945 designs, and knocking rockets together with bits of old metal. Its' not glamourous, but they are still luanching multi-million dollar satelites into space.

Agile Development - Learning from Tommy Hilfiger

2005-07-25T05:31:00.000-07:00

I've allways seen software development as more of a creative process than a science. A couple of recent experiences has convince me even more of this.

The first incident occurred when I was explaining story writing to a BA. "Oh" she said, "a story board." "It sounds just like the story board we'd put together when working on a new fashion collection." Prior to being a BA she worked for several years in fashion. Apparently, in a fashion boutique someone would come up with a concept for a new range or collection on a story board. A team would then populate the board with ideas: drawings and snippets of fabric building and expanding on the basic theme.

More I thought about her analogy, the more I liked it. After all, in the beginning, the concept for a new piece of software is no less creative than a fashion concept. Stories are nothing more than incomplete snippets that help flesh out the basic vision. Some stories will be accepted, depending on their percieved value, much like ideas in fashion.

The second incident occured whilst watching television. The programm was called "Rich Girls", and followed the life of Tommy Hilfigers' daughter. Tommys' team had come up with a story board for a collection aimed at teenagers. As a way of proving their ideas, Tommy asked his teenaged daughter and one of her friends to come into the office and go through the story board.

His daughter took her "work" seriously, and went to efforts to explain to her friend that "she needed to be honest." If she didn't like something she should say so. "Don't be diplomatic". The scene was an interesting one. The story board taking pride of place in the centre of a large room. Fashion designers huddled in a door way peering in. Tommy leading his daughter and her friend through ideas on the board. Tommys' focus was soley on his daughter and her friend, after all they where the potential customers. The fashion designers were tense, and uncomfortable, as they listened to a severe critique, handed out by two teenagers.

Tommys management technique was interesting. There was only two experts in that room that day, the teenagers. Watching them say, "I wouldn't wear that, or I wouldn't be seen in stripes like those", with everyone listening was amazing. A real lesson in direct customer feedback.

I recently read a post by Rachel Davies where she refers to the lies and half truths that characterise the relationship between development and the business in most organisations.

Tommy Hilfigers management technique seemed miles apart from what I'm use to and what Rachel describes in her blog.

Managing a creative process is different to managing a defined science. Perhaps IT managers could learn a lot from the creative professions. Acknowledging that software development is essentially creative, would be a good first step.