To the Victor the Spoils

Yardena kindly supplied this link to an interview with Erik Meijer, Gilad Bracha and Mads Torgersen, talking about Programming language Design and Evolution. The interview was interesting with Gilad confirming something that I have suspected for a while, which is that state is optional, and that most/all? problems can be decomposed into immutable functions. The problem with pure functional programming though is that it is really hard to do well, simply because we just don't think of the world this way. We think of the world as having state and we find it easy to model the world as stateful. Gilad uses the example of the interviewer as being a thing that is "stateful", rather than being a sequence of immutable things each one with no prior memory, only existing for a given moment in time.

Given that programming is hard enough as it is, relaxing the constraint of "no state" and introducing the idea of encapsulated state is a big leap forward I think, and represents a great, still untapped opportunity when compared to "anything goes" procedural programming. Contrary to the "OO" marketing hype over the last decade, the dominant programing paradigm today is still procedural in my view. The direct consequence of procedural based hybrid languages like C++, Java and C#, that only pay lip service to OO and functional ideas, whilst lacking the uniformity to be truly functional or Object Orientated.

It was interesting to hear John McCarthy say that the idea of encapsulated state, as introduced with Smalltalk has been the only significant step forward in programming language design in the last 50 years since Lisp. It was painful to watch the interview as the interviewer made claims for languages like C++. Even going as far as claiming that Mac OS X was written in C++ (although OS X is written in C and Objective-C), and that we still need C++ for all our "Systems" programming needs today. Gilad winced more than once. Which prompted the question: "Why is it that "beautiful" languages aren't more popular if they are so much better?". Gilad tried to answer this one, using a form of Alan Kays "prior context" argument, saying that programmers didn't like new ideas, and that change needed to be introduced incrementally.

Interestingly it was the C# guy Mads, who debunked this half truth, by saying that programmers were interested in new ideas, but that "new thinking" didn't pay the bills. Mads also admitted that a lot of the work that lead to the dominant static VM platforms of today was based on wrong thinking stemming from compiler based, code optimisation ideas, and it would have been much better to have started off with a dynamic platform and to have layered on top an optional type system, much like Gilad advocates. He puts this realisation down to hindsight, forgetting to mention that people like Alan Kay and Dan Ingalls were advocating just this at the time.

The biggest disagreement was over the future role of C++ with Gilad claiming that the language was as good as dead, and that they should have stopped extending it long ago. The others, including Erik, felt that it was still useful with advances in static analysis tools still being made today. It was painful for me watching advances in IDEs and Operating Systems being attributed to C++ and it's prodigies Java and C#, when it is clear to anyone with any grasp of history that we owe the modern Graphical, Windowing Personal Computer world to Smalltalk. You could see the pain on Gilads face as he sat through this mis-representation of history, mostly by the interviewer.

Mads made the point, that the successful languages today are all incremental changes to the thing that existed before them, namely C. And I guess this is the rub. C/C++, Java and C# are all successful. Smalltalk is not. To the victor the spoils, which means that the victor gets to write history.

This is fine with me as long as Smalltalk gets to write the future :)

Newspeak - First Impressions - Underlying Beliefs

First thing to say, after watching Gilads presentation on Newspeak, is that I feel that I'm not qualified to deliver a rigorous critique. This in itself raises an interesting question. After 18 years software engineering experience my "education" is insufficient to evaluate a new OO language that builds on ideas that are over 20 years old. Why? Well, my professional career (the usual diet of Java APIs and frameworks) hasn't prepared me for Newspeak. In fact it is only my self education which I performed at home in my spare time that allows me to grasp most of what Gilad is speaking about. This telling fact says more about our industry I believe then it does Newspeak.

Beliefs and Motivation
You may find beliefs a strange place to start an evaluation of a programming language, but I think that the beliefs of the protagonist behind a language lay at the core of what a language is about, and whether it will appeal to me. Language design, like any other form of design is about trade offs, and it is your beliefs that determine the relative value you place on competing ideas and principles. Well the first thing to say is that Gilad is still a Smalltalker at heart, and he describes Newspeak as a direct decedent of Smalltalk. The other significant influences he mentions are Self, Beta and Scala in that order of significance. Scala seems like the odd man out here, but when you realise that its influence is much smaller then the others (the formalisation of object constructors, which is something Smalltalk never did), then the idea of Newspeak as a successor to Smalltalk makes sense.

My concern was whether Gilad believed in pandering to the needs of the tool over the beauty of the language from the perspective of the programmer. Well he makes his feeling clear in is preference for:

id = letter, (letter | digit) star. (Newspeak)

in comparison to:

id = letter().seq(letter().or(digit()).star(); (Java)

When attempting to express the BNF Grammar:

id = letter, (letter | digit) *

He refers to the redundant parenthesis in the Java version as "solution space noise". He says that what we should be doing is expressing our problem. Stuff in the code that is there to satisfy the tool (Compiler) is noise and should be reduced to a minimum. This belief sits fine with me. It also fits with my artist analogy. An artist is more interested in his subject then he is in his tools. Languages like C++ and Java just miss this point, and when I think about it, this is the central idea I learned from my first encounter with Smalltalk all those years ago. Just going through Squeak the other day I stumbled on things like this:

(2 + 3i) sqrt.

What do you think this does? Yes, you guessed it. It answers the square root of the complex number 2+3i. Now imagine expressing this in Java?

OK, if Smalltalk is so beautiful what is left to be solved in Newspeak? Well plenty actually. In terms of beauty, Smalltalk has its warts too. Like having to place the word "self" before each message send when sending a message to your self. Newspeak, like Self makes the receiver as "self" implicit.

The previous BNF grammar expressed in Smalltalk is:

id := self letter, ((self letter ) | (self digit)) star.

Which is more pretty then Java but not as pretty as Newspeak.

Smalltalk also lacked the ability to define encapsulated abstractions that were larger than a single Class. So although Smalltalk is uniform and "turtles" all the way down, when it comes to Classes everything is horribly coupled in a single global hashmap. Newspeak solves this lack of modularity problem too. So Newspeak is turtles all the way down and all the way up as well, leading to pluggable libraries etc. It achieves this by borrowing the idea of nested classes from Beta. So a module or a library is a class that contains other nested classes. I'll discuss this feature in more detail in a future post, but it is very significant and Gilad highlights it as the major difference between Newspeak and Smalltalk. It allows you to swap out abstractions at a category or library level with Newspeak without worrying about ugly inter-class dependencies (coupling). So Newspeak can be shrunk as well as extended unlike Smalltalk.

I've said that Smalltalk is turtles all the way down. Well that isn't quite true. All OO abstractions (turtles) should encapsulate their insides from the outside world. Smalltalk does do this for member variables, but in Smalltalk-80 all methods are accessible from outside the class in which they are declared, including methods that are meant to be private. When I first came across this, I thought it was a minor oversight that would be fixed soon, but interestingly the situation is still the same today in Squeak. Gilad intends to fix this oversight in Newspeak. So one of Gilads motives seems to be to finally address the design flaws in Smalltalk.

I find that many of Gilad's throw away ad-lib comments are more informative then much of his precise technical explanations. Throughout the presentation, and especially when he would switch to his Newspeak IDE, Gilad would make reference to the past failures of Smalltalk, and how he hoped to address those issues in Newspeak. In the eyes of many, Smalltalk never really made the transition from a research project to an industry ready software engineering platform. Strongtalk was one attempt to complete that transition. Strongtalk dropped the "many windows" class browser approach of Smalltalk-80 and adopted a collapsible pane approach that would be familiar to users of Eclipse today, and feels more natural to people coming from a "file" based metaphor for organising classes. Newpeak does the same. Thankfully Newspeak drops the type annotations of Strongtalk, with Gilad admitting that many of the late bound scenarios that Newspeak is designed to address would be almost impossible to verify statically. Gilad also plans to have a much improved FFI (foriegn feature interface?) for Newspeak, when compared with a typical Smalltalk dialect like Squeak.

Smalltalk as always been an island. Once you were on it the weather and the scenery was great, but you couldn't bring anything with you from the outside world except your bathing trunks, and you found yourself missing a bunch of stuff you had gotten attached to. In contrast Ruby fits into the Unix ecosystem like a dream, and you can use almost any windowing system, foreign language library and API available under Unix(/Windows/Mac OS X) from within Ruby. You can also "shell out" to a Ruby script from a Unix program too, so their is two way communication. Gilad spoke about a similar ambition for Newspeak with a FFI that allowed callbacks from foriegn functions, although he didn't go into this in detail. He does mention the use of Actors as a means of inter-process communication, so perhaps this is a pointer to how some integration may be done.

So to sum up, Gilad believes in clearly expressing the problem over assisting the tool (Man over Machine) and uniformity and modularity ("turtles") all the way up as well as all the way down. He is motivated by creating a successor to Smalltalk-80 that complies with sound software engineering principles and interfaces well with the current incumbent technology ecosystem. His aim is to create a language which is both esthetically pleasing and software engineering sound. The end goal is an elegant and consistent software engineering platform for use in industry.

I was going to give my impressions on how well Newspeak in its current incarnation actually delivers on these lofty goals, but this post is already long enough. Besides you can watch the presentation for yourself :) I'll be coming back to Newspeak in future posts, where I intend to pick up where I've left off here.

Update 29th April 2008
Gilad has kindly offered the following corrections and clarifications:

Re: your review. Well thanks! A quibble or two:

1. The basic FFI with callbacks is working; the higher level convenience layer isn't there yet. We are just calling to/from C (or Objective C) at the moment. Using actors to interface to the outside world was Alan Kay's original plan, but it was far ahead of its time.

2. Types. I haven't given up yet. It's just our lowest priority and a very hard problem. I would like to support pluggable types in Newspeak, and we currently support and encourage type annotations even though they are not checked.

Making Great Music

Thinking out load on the web is a risky business, but it has it's benefits. With a little help from your friends it can lead to a deeper insight. The logic of the case made by Yardena and Andrew in response to my original post on Man versus Machine made sense, but it still left me feeling a bit uncomfortable.

Had my artistic analogy run out of steam?:

I love reggae music. I grew up on it: Bob Marley, Dennis Brown, Gregory Isaac, the reggae greats of the 1970's. In recent times Reggae as become more commercial with modern"artists" placing less emphasis on spirituality. I find this change rather regrettable, because without spirit the music becomes just noise.

Then I got to thinking more about those old Reggae classics. If you listen to the original vinyls the music is punctuated by pops and clicks, and even drop outs in places. Sir Clement "Coxsone"Dodd, the owner of the Studio One record label had humble facilities at his recording studio in Kingston Jamaica. I believe many of those originals were recorded using nothing but a simple eight track in a small unsound proofed room.

Some of those classics have been remastered, which gets rid of the pops and clicks, but sometimes you loose a bit of the feel too. Don't get me wrong those old classics are still great pieces of Music, but imagine how they would have sounded if Clement Dodd had the benefit of Modern Recording facilities back then? Great Musicians and Singers with great tools. Hmm...

The Answer: Man and Machine in Unison

In my last post I proposed the question. What is more important the Tool or the Man when it comes to software development? It was in response to Gilads attempts to eliminate certain dangerous "features" from programming languages. I questioned whether this was the right focus and I got some interesting responses.

This one in particular rang a bell:

Rather than looking at it as Man against the Machine, I see this as a challenge of building a bridge between the two, giving both sides respect and attention. I agree with Andrew - if the machine can't solve the problem it should delegate to the person, but do it simply and clearly. In a more philosophical sense, I think because computer science is so young, we have not figured out properly how to transition from "science" to "engineering" yet. To do this we need tools that are controllable and safe alongside creating good programming curricula to train "the pilots".

I think both Andrew and Yardena are right in their responses. It isn't necessarily one or the other, Man versus Machine. We need both working in unison.

Who rules - Man or Machine?

I've been following Gilad Bracha's blog on programming language design. The concepts Gilad touches on are profound and his blog is a real interesting read, I recommend it.

Some of Gilads most recent posts have left me pondering whether he is focusing on the right things? Gilad calls himself a Computational Theologist. This is an interesting title since theology has to do with beliefs and belief systems. From Gilads posts on Monkey Patching and Cutting out Static, I get the feeling that Gilad believes that the machine should 'help' the programmer to do the right thing. What beliefs that underpins such an assertion? Perhaps such a believe stems from an acceptance that most programmers are poor to mediocre and need all the help they can get? Maybe this believe stems from an idea that a computer program is more a piece of mathematical logic then a piece of creative art?

I am assigning a lot of beliefs here to Gilad, which are probably things he doesn't believe. The point remains though, at a fundamental level we have the choice of either believing in people or believing in machines. Andrew has a post on Smalltalk where he refers to the original Smalltalk Byte article. The Smalltalk researchers had a human powered approach to 'computer research'. Reading their paper it is clear that they believe in people and our inherent creative abilities. To them the machine is merely a tool. Man, the creative artist exploits tools and mediums to express himself. A computer is merely one such tool with a unique set of characteristics. The purpose of a programming language is to allow Man to exploit the Computer.

It was interesting to hear John McCarthy say a similar thing. For example he rubbishes the commonly held belief that "goto" is evil and should be banned. As an artist I'm not sure how I feel about people banning things and limiting my expression. Imagine a word processor that didn't allow you to use certain words like "fuck" or "piss"? There is a strong argument that such words aren't the most effective form of communication , but can we say they should be banned in all instances?

There is an interesting point of debate here, and I have no firm conclusions, other than to say that people have far more potential then machines. At best machines can help people to explore their full potential and extend their influence and reach. But a machine has no consciousness, no intelligence, no imagination. A machine has no spirit.

As humans we have two parts to our brain. A logical side, and an emotive artistic side. I believe that both sides are involved in everything we do, including programming, and we need to appeal to both. This means that the most correct and safe programming language may not necessarily be the most useful.

I love reggae music. I grew up on it: Bob Marley, Dennis Brown, Gregory Isaac, the reggae greats of the 1970's. In recent times Reggae as become more commercial with modern"artists" placing less emphasis on spirituality. I find this change rather regrettable, because without spirit the music becomes just noise.

What Really Matters?

My posts of late have been pretty opinionated and uncompromising. Why? Well as a Software Industry we are pretty good at creating reasons to do a bunch of stuff that doesn't really matter. I guess doing this other stuff is easier then tackling the difficult task of doing what does matter.

Anyway, I stumbled on this Quote on the C2 Wki:

What really matters?

Software is too damned hard to spend time on things that don't matter. So, starting over from scratch, what are we absolutely certain matters?

1. Coding. At the end of the day, if the program doesn't run and make money for the client, you haven't done anything.
2. Testing. You have to know when you're done. The tests tell you this. If you're smart, you'll write them first so you'll know the instant you're done. Otherwise, you're stuck thinking you maybe might be done, but knowing you're probably not, but you're not sure how close you are.
3. Listening. You have to learn what the problem is in the first place, then you have to learn what numbers to put in the tests. You probably won't know this yourself, so you have to get good at listening to clients - users, managers, and business people.
4. Designing. You have to take what your program tells you about how it wants to be structured and feed it back into the program. Otherwise, you'll sink under the weight of your own guesses.

Listening, Testing, Coding, Designing. That's all there is to software. Anyone who tells you different is selling something.

-- KentBeck, author of ExtremeProgrammingExplained

If I had my way these four points would be prominently displayed in every IT Department out there.

Doing It Twice

As an Agile Coach I have found it somewhat disheartening the way Software Development Organisations choose to cherry pick Agile practices that fit in with there current beliefs and culture. I mentioned this in my "Me too Agile" post.T

Interestingly it is not the first time this has happened. After hearing several reports that Winstons Royce original paper on Managing the Development of Large Systems proposes a very different approach to software development then the way "Waterfall" is has been interpreted.

It turns out that Royce understood the importance of feedback and Emergent Design too:


STEP 3: DO IT TWICE
After documentation, the second most important criterion for success revolves around whether the product is totally original.If the computer program in question is being developed for the first time, arrange matters so that the version finally delivered to the customer for operational deployment is actually the second version insofar as critical design/operational areas are concerned


If you replace the word documentation with communication then there is nothing here that would look out of place in a modern paper on Emergent Design. It is a shame that people chose to cherry pick back then. Taking the bits that were convenient and leaving the bit that weren't.

Skimming through the paper, there are a few eye openers that show that nothing is new in Software Development, we just choose not to listem. This was one of the more surprising ones that jumped out at me.

Architects - Who needs them?

William as responded to my post on "Me too Agile" and his response made me think that my post required further explanation. An Architect as a software development role is a relatively new idea, appearing in the mid 90's along with the boom in 'OO' technologies and middle ware.

Before this the only architects I was aware of produced paper drawings for buildings. The architect would design, and builders would construct. The term Architect is borrowed from the building trade. Its roots in the building industry are quite revealing I believe and say a lot about the assumptions, philosophy and organisational cultures that led to its use in Software development.

It seems strange to have to say this, but in the early days of software development, there wasn't roles like Project Manager, Business Analyst, Systems Analyst, Architect, Test Manager etc. There was usually a couple of guys who sat down and decided that they wanted to write a computer program. When Thompson and Ritchie decided that they wanted to write the Unix Operating System they didn't have an Architect. They did the design and wrote the code themselves. They even went as far as producing their own programming language, so they created their own tools too. As a team they were self reliant, with a single focus the creation of a working program.

The purpose of my post was to question the relevance of the role of architect in an organisation that subscribes to Agile values. William in response quoted the Agile manifesto as a definitive statement of Agile Values. Well before the term Agile was coined, the category of software development methodologies we now call Agile went under the name light weight processes. What all these methodologies had in common was a rejection of an high ceremony approach to software development. Approaches like OMT, Objectory, RUP (which now coincidently claims to be (me too) Agile :)) that prescribe a number of distinct development phases with a number of distinct roles, handing off work to each other through a number of concrete intermediate work products. 'Light weight' meant getting rid of as much of this as possible and getting back to a way of working that would be more familiar to Thompson and Ritchie.

This view of the world, builds on a management philosophy that says that decision making should be delegated to the lowest level possible within an organisation. In short the people who do the work are best placed to make the right decisions. This idea of worker empowerment builds on the work of Edward Deming, and reaches a pinnacle in the Toyota Production System as described in the writings of Taiichi Ohno.

In keeping with this philosophy, the Japanese have developed a number of approaches to new product development, which we commonly refer to as Lean, Just-in-Time, Agile, etc. One of these approaches is deferring design decisions to the last responsible moment. The reason for this is to allow concurrent design. Toyota do not have Architects. They have a chief Engineer who champions the project and as an overall vision, but he does not make design decisions or tooling decisions. These decisions are delegated to the design teams who are autonomous. These teams try to keep their options open avoiding looking themselves into decisions which are not easily reversed later. The reason why they do this is because they do not own the whole design. For example the boot design team may need to change their design late into the development cycle. If the team responsible for the interior space of the car decide that people in the back really need an extra inch legroom to compete with the latest competing models in their target market segment. Then the boot design team need to be in a position to respond quickly, by changing their design reducing the boot space to accommodate the larger interior. This agility is why they defer irreversible design decisions.

Simon's paper on architecture, although it doesn't use the word "Agile" explicitly, makes much play of this Agile principle. The point I was making is that this principle is part of a wider philosophy, and that philosophy is not consistent with the idea of Architects and Architecture as borrowed from the building industry.

Taiichi Ohno would not recognise the use of deferred decision making in a context where the architect was the technical authority yet not responsible for doing the actual work. What we have here is a mix of two separate belief systems. The belief system that says that decision making should be centralised stems from the Management teaching of Fredrick Winslow Taylor and his idea of Scientific Management. A good example of this philosophy in practice was Henry Ford and the Ford Company. The Japanese came to the US after WWII and looked at the Ford production System and largely rejected it as inefficient and wasteful.

The point I'm making is that these are two separate contradicting belief systems, based on different values. We can question the relative merits of each, but cherry picking practices from one belief system and applying them out of context into an environment with the opposite belief system is not likely to produce the intended results. The beliefs come first and the organisational structure stems from those beliefs.

So whilst it is true that Simon did not appropriate the label Agile, he did attempt to reconcile the role of Architect which is essentially a taylorist construct, with the principles of Agile product Development which stem from the beliefs of Deming.

This is the background to the point I was trying to make.

Generics - An OO Anti-Pattern

I'm obliged to use Java 1.5 at my latest client. One of my gripes with Java is that it doesn't encourage an OO programming style. As a Coach, I tend to find that most Java programmers lack a full understanding of OO design principles. In fact I can count on one hand the number of Java programmers I've come across who have an understanding of OO which is at least as good as mine.

In contrast all the Smalltalk programmers I've met, understand Objects very well and I'm sure most could teach me a thing or two. So Sun decided to revamp Java, a supposedly OO language. You would have assumed that they would have borrowed even more from Smalltalk, but no. Instead we get Generics. So why?

Before answering this question. I should really spell out why Generics are not compatible with OO design principles. Objects are meant to be loosely coupled runtime entities. Objects do not exist at compile time, they come into existence when you run your program (or with Smalltalk, they come into existence the moment you load your image). Objects should hide both their state and implementation from others, this is how they achieve low coupling. All that is exposed is their message interface. In Smalltalk, this interface is known as the Object's Protocol. So to communicate with an object, and get it to do something useful, you need to know its protocol and nothing else.

OK. Lets compare this with Generics. Firstly in the Java view of the world, message sends are replaced with virtual functional calls. A Function call as a way of sending a message to an Object is only possible in Java if you know the type of the receiver. In Java the receivers' Type is either a reference to its implementation (its Class) or to one of its implemented Interfaces. So straight away the idea of hiding knowledge of the implementation through message sends is lost. So along comes generics. Does it improve matters any? Well no, in fact it makes things worse, a lot worse.

If the answer (return value) to the message sent to the receiver is a collection, then the Type of Objects that it may contain is also part of the message interface in addition to the Type of the Collection (container) itself. So Generics leak a lot more information about containers. It gets worse when you consider subclassing and method overriding with generic containers. The complexities of what should represents a valid answer to the same message is mind boggling.

So generics leak information like a sieve, and break the basic OO tenant of information hiding (encapsulation) and low coupling. Information hiding is useful, because it allows you to substitute objects at runtime. The substitute could extend the protocol of the original, or implement the same protocol in new ways with different side effects. This idea is what is commonly known as polymorphism, and gave birth to the idea that OO programming could lead to components and re-use.

So back to the question Why? Information hiding is powerful when it comes to malleability and extensibility, but for some perhaps it is too powerful. In Smalltalk to know the Type of an object you need to send it a message. There is no other way. The Type is not manifest in the code. So the Smalltalk IDE is a running Smalltalk program containing a bunch of Objects. Classes themselves are Objects and to know what Type an Object is the Class Browser Object sends it a message to which the answer is the Objects Class. So with Smalltalk you only get to know anything about an object once it is running. Because of this the Smalltalk environment is always alive, always running, all the way through the programming cycle. The Smalltalk image, when loaded contains both IDE objects such as the Class Browser and developer Objects such as application Classes. The Browser sends messages to Class Objects to reveal their methods, and to method objects to reveal their source code. Programmers edit the code, and then send a message to the Compiler Object to compile the method. None of this is possible without running the image.

This approach doesn't help you if you don't like running the code to find out what it does. If you want to perform a static analysis you need more information at compile time. So using Generics is a way of providing this information removing the need for dynamic casts. So why would you want to get rid of casts? Casts are one of the gaps in your ability to fully analyse your program statically. A cast is an explicit admission that static analysis can't help in some scenarios, and you still need to defer some type checking until runtime. Generics is an attempt to bridge this gap, in an attempt to provide complete type safety at compile time.

For programmers who like their code to tell them what it will do before they run it, then this meta-data laden declarative approach is viewed as a benefit, but as Steve Yegge points out, programmers should not need such "training wheels". To know what a program does, what you should do is run it (test it). This unnecessary meta-data obfuscates the code and limits the degree to which the code can be deemed fully Object Orientated.

True Object Orientation relies on late-binding, which occurs at runtime. The whole point is that "you don't know for sure" what it is you are sending a message to. Allowing the receiver to be substituted. Manifestly stating that you know, limits polymorphism and artificially restricts the computational model.

Pattern Languages and Painting by Numbers

Steve Yegges post on Noobs has really got me thinking about models and the purpose they serve. I think that models help fill a void in the design space. To begin with all we have is a problem and a blank canvas. A skilled artist will look at his subject and start sketching perhaps at first with a pencil. He will then refine and re-work his sketch before moving on with oils.

To the novice artist the blank canvas must be daunting. I have noticed that novice developers with a poor understanding of OO find OO programming from first principles a daunting task too. They are much more comfortable starting out with a template or a framework. A standard sketch. So if the subject is a web page, then the Struts framework which implements the model-2 web application pattern is a comforting place to start. All the novice then has to do is colour in the details. Just like painting by numbers.

When coaching, I have had an uncomfortable feeling about patterns. Novices are great at learning patterns by rote, not so good at knowing when to apply them. And even when they do apply them correctly, few seldom know how to tailor such patterns to create a novel design themselves. Worst still is where someone else has imposed a pattern and the programmer follows it religiously not noticing that the pattern doesn't quite fit for the problem at hand.

Artist do not paint by numbers. Sure they learn a lot of styles (patterns), but ultimately their creations are there own. So how do they create and come up with something novel when faced with a blank canvas? Well they rely on first principles. They understand perspective and how it works. They understand light and how it works too. They understand their medium: oils, canvas, brushes and strokes. They have a deep understanding of their subject aswell. Bone structure, muscles, skin tone etc. They also have a good eye for detail and can see things in their subject of interest, drawing out important nuances, and giving them prominence in their final piece of work.

Sound technical understanding based on first principles, a catalogue of prior work (patterns) for inspiration, a keen eye for significant details, and a well honed instinct for creativity. These are the quality of a good artist. They are also the qualities of a good programmer too.

I might be over ambitious, but I believe that average programmers can obtain this level of skill with coaching and practice, and model obsession as we have come to know it is not needed and can be replaced with something better. In a series of blogs I hope to strip away standard patterns and get back to first principles in an attempt to show the cost we are enduring by painting by numbers. I also want to show that going back to first principles isn't that daunting and can be both beneficial and a lot of fun.

Me too Agile

William Martinez has pointed me to a paper which talks about Emergent Design from an Architects perspective. Apparently Architects are now deferring architectural decisions to the last responsible moment in an attempt to be more Agile.

On the same site they present a skills matrix defining the role of an architect. looking at the matrix it looks as though the new Agile Architect is a cross between a Systems Design Engineer a Senior Development Lead and an Agile Coach.


To me the term Agile Architect is a misnomer. Architecture as I understand it is about technology selection high level design. In an Agile team, the team are responsible for making such decisions. If the Architect is part and parcel of the team or is an outside consultant that the team can consult then I guess that this approach is still consistent with say Scrum. But ultimately the team must decide. Since the team is ultimately responsible for delivery. Anything else would brake the central tenant of Scrum which is that the team should be self organised and make their own decisions.

It puts me in mind of Scott Amblers website on Agile Modeling. Again a central tenant of Agile Development is feedback. Yet you don't get much feedback from a non-executable UML Model until pretty late in the development process. So again Agile Modeling is a bit of a misnomer.

So why are we seeing such things? Well in my opinion this is the consequence of organisations that have structured themselves around a waterfall "production line" view of software development who never the less like the sound of Agility, yet do not want to confront the need for organisational change. To me these are just symptoms of Agile as a fad. Wannabes jumping on to the bandwagon. Me too Agile.

True Agile seems to be taking root in environments with less cultural baggage such as new startups. But for those who are merely interested in trying out the latest fad without making a real commitment to cultural change, they can change their practices a tad and re-brand themselves as Agile. "Me too" Agile. Which is what I believe we are seeing here.

SOA and the N-tier Fallacy

My last post on Emergent design got me thinking about other common architectural patterns (in addition to ORM) which may not necessarily be the best solution for the types of problems to which they are commonly applied.

I believe N-tier architecture where N=3 is a common choice, is yet another example of Big Upfront Design locking us into a unnecessary straight-jacket even before we look closely at our problem. Now the purpose of each tier in such an architecture is to provide a service interface to subsequent tiers. The goal being that tiers are loosely coupled and hence inter-changeable. From a clients perspective the tier you will most likely want to change is the presentation tier. So a PDA client will want a different presentation tier then a desktop client, for example. But both clients would want to share everything else from the service tier down.

All of this is common wisdom and as spawned a whole architectural vernacular of its own around the term Service Orientated Architecture. So what is wrong with SOA you ask? Well nothing, other than the term is not very specific and it tells you nothing about the type of problem to which it applies. SOA is the proverbial solution looking for a problem. So lets start this discussion again from the other direction. Lets choose a concrete problem and then identify a clean working solution.


The problem: My users want to store a bunch of related data records in a database. They also want to create, retrieve, update and delete (CRUD) these records remotely through a web browser interface .

OK. They want to store related data records. Sounds like a job for a relational database to me. They then want to view these records over HTTP. Well the verbs in HTTP (POST, GET, PUT, DELETE) sort of map to CRUD operations so why not make use of this? Also my user will view data records via HTML web forms presented to the browser. So there is inherent data coupling between my web presentation and my entity/domain model. Why not except this coupling as intrinsic?

From an Emergent design perspective, the simplest thing that can possibly work is an architecture that allows you to create data record hashmaps from HTML presentation views, and persist them in a relational database. Better still from an OO perspective, it would be even better if those hashmaps could persist themselves, and perhaps do other useful house keeping like data record validation that only they should know how to do. Viola - The simplest CRUD architecture possible. A Controller to handle the HTTP verbs and map them to actions on our database. An ActiveRecord that models database records as an hashmap and is capable of persisting itself, and a View that translates the model to an HTML presentation ready to be sent back to the browser over HTTP.

Sounds very much like Ruby on Rails to me. Now how do you think DHH got there? Did he use big upfront design, or did he choose to focus on his problem afresh and apply emergent design principles? Well it should come as no surprise to find out that Rails was harvested from a real word CRUD application that evolved over time. It should also come as no surprise either to find out that 37 Signals are big advocates of TDD. In fact Rails is the only web framework I know that has TDD built in.

One last thing. The astute amongst you are probably asking what happened to our nice separation of concerns and SOA? Well if you look at my problem statement I didn't need it (YAGNI). My client was using a web browser. I've saved myself a heck of a lot of unnecessary code and shipped my release 3 months early. Its now out there making a ROI for my customer whilst those N-tier folks are still deciding how many tiers they need. Hang-on I hear you cry. So what happens when your customer does decide that he needs another client to the system:

Problem2: Not only does the system need to support CRUD data entry by people, but in addition, we have recently gone into partnership with another company who wants to access the same data records using a remote system. So we now need some way to support this too.

The world as changed so we need to change our system to suite. Unlike the BUFD guys we will only make such an investment if driven to do so by our problem. OK. We understand how to map HTTP verbs to database actions, we also know how to map our model to HTML. Given this it makes a lot of sense to re-use this infrastructure and choose a service interface that makes the most of HTTP, we settle on a RESTful approach. Our previous CRUD based controllers still apply, but now the presentation needs to be machine readable XML instead of HTML. We could put a lot of if.. else.. logic in our controllers to determine what type of presentation to create, but we decide that this would represent a lot of code duplication, so we choose to encapsulate this logic in one place which all controllers can use. This approach is so successful that we then decide to factor it out as part of our home-grown framework. Sounds pretty much like how Rails 1.2 was born.

So now we have a Restful Application that is fully SOA aware and is built on our existing technology base. The only new technology is XML, and given that XHTML is a variant of XML, we could argue that we haven't had to introduce any new technologies at all. So we have managed to introduce a Service tier when we needed it at very little cost.

Some following this discussion may come to the conclusion that DHH just got lucky. To them my answer would be that you create your own luck. By gaining a deep understanding of the problem, keeping the design clean and simple and deferring unnecessary design decisions rather than prematurely locking yourself into false assumptions; then 9 out of 10 times evolutionary paths will present themselves without you having to plan (guess) for them upfront.

In the 1 out of 10 cases where there is no low cost option, then what your problem is telling you is that your application requires diverse personalities, and whether you plan for this upfront or evolve it later, producing such an application will be expensive. The benefit with emergent design is that by applying the YAGNI principle, you will not incur this cost unless you really need to.

Emergent Design

William Martinez as an interesting repost to Cedric Beust's recent criticism of TDD. As always William as something interesting to say, and he makes the point I made on Cedric's blog, that TDD is no the whole design picture and you need to use a different design "language" at different levels of design abstraction.

Given that William is an Architect, I seized on the idea of using "different" design languages and what I see as the over use (misuse) of modeling languages such as UML. I tried to express this in a comment on his blog, but for one reason or the other (probably due to its length :^)) his blog wouldn't except it.

Thinking it through, I believe what is central to Cedric's views on TDD is a lack of awareness of Emergent Design and how it works. It would be interesting to see Williams take on Emergent Design since it gets rid of different design roles such as Architect, Analyst and Programmer combining them into one. The way it should be :)

Getting rid of Architects? Provocative I know. Anyway I include my comment here as a description of Emergent Design. A warning. It contains plenty of strong opinions :)

Hi William,

I sort of agree. We have spent a long time trying to place "good code" in a box. You can't legislate for "good code" yet we try. The design layers you speak of are one way of stratifying code into deferent levels of design abstraction in an attempt to legislate and box up Software Development. Yet it is the code that determines the correctness of a program and ultimately the success of the design. Once we arrive at the code, all other models are merely a projection of the one true model, the code model. Other models have no concrete existence.

So how do we test the quality of our code model? Well it is like the Shrodingers Cat paradox. We do not know whether our design is correct (works) until we execute the code. We do not know whether the cat is dead or alive until we look in the box.

I think an over emphasis on modelling can lead to a situation where we forget this basic truth. Second to working code, we would also like our code to be clean. How do we know our code is clean? Well for OO systems removing code duplication through inspection works well. TDD punctuates the coding process allowing you first to check for correctness through verifiable computation (a green bar), then inspect and safely remove code smells such as duplication, coupling, poor cohesion etc with the safety net of knowing that you aren't breaking correctness.

Using this approach, your design can emerge empirically. I agree that this bottom up approach to design has its limits. It can lead to locally maximised solutions (micro-design). So how do you find the global maxima (macro-design)?

XP practices come to our aid here too. By implementing the solution one customer prioritised story at a time, you are focused to extend your macro design incrementally, the simplest way that can possibly work. So at the story level your macro design is emergent too. But that leaves one last question. Each story's implementation needs an architectural context. How do you decide on your architecture? Well other than Kent Becks "Metaphor" idea XP provides little guidance here and we are back to guessing.

Unlike traditional design though, we acknowledge that our chosen architecture is just a guess and is yet untested by our requirements (stories). Given this we make a minimal investment in our untested architecture. Some people call this a walking skeleton. As we add flesh to the bones, we may find out that the bones are in the wrong place or are the wrong size, so we change them, and in this way our architecture emerges too. We may end up with the skeleton of a Horse that emerged from the skeleton of a Pig say, all driven by our requirements and an overriding drive for simplicity.


How do you describe this process in a book? The answer is you don't. This process requires a great deal of skill and judgment applied at each punctuated phase/cycle. Top designers like Kent Beck design this way, lesser designers can learn this approach by example and through experience and practice. The idea of learning from a Teacher and developing your skills through practice is not revolutionary and is the norm for most creative professions. Unfortunately we have managed to fool ourselves into thinking that programming is not a creative profession. So we do not have the teachers and coaches, and we do not provide newbies with the concrete feedback they need to learn from their experiences. If a programmer ends up having to code around your architecture and never gets an opportunity to communicate this to you, how will you learn and improve?

It is true that not all modelling can happen at the keyboard whilst writing tests/code. But it is possible for a lot more modelling to occur this way then people think. I would say that as much as 90% of design time can be spent at or near the keyboard, and the other 10% can be ephemeral (back of a napkin, white board session, CRC cards etc). I would also say, that modelling at all levels of abstraction should be done by the same person. Remember that code is merely a human readable representation of the computational model, and it is the computational model that is our final program.

So how do you get here? Well firstly it needs to be experienced to be believed. You end up with a design that actually fits your problem, rather than a design that you thought would fit your problem, and got locked into early on. Have you ever got to the end of a project and thought if I knew at the beginning what I know now I would have designed it completely differently? Well with emergent design you get to change the design every time you learn something new. No need to guess during a BUFD (Big Up Front Design) phase.

An example of BUFD (IMO) is how we all have got locked into ORM, when it may not necessarily be the best fit for the type of problems (CRUD) we are trying to solve. I think we have become model obsessed. Steve Yegge has an interesting blog post where he explores this further. I think Steve is right.

We need to stop aping other professions and spend more time mastering our own. Software Development is a unique craft that calls for a special set of skills that cross a number of layers of abstraction. Unlike the building trade it doesn't lend itself to stratification. We need to be able to program, design, architect and analyse all at the same time in small micro cycles each lasting as little as a few seconds. We need to punctuate the process with concrete feedback. At each cycle we need to focus on a specific problem one at a time. Our cognitive abilities are finite and we must divide and conquer, so as not to overload them. And we need to allow our design to emerge, deferring design decisions to the last responsible moment, where we have learned the most and are best equipped to make the right decision. We also need to recognise when to break out in the light of new information, and reconsider the big picture, either as an individual or as a team exercise.

These skills may not fit neatly into the stratified organisational structures of our current software development organisations. But the Uber programmer doesn't fit into this world either. We need to be producing more Uber programmers who are creative, highly skilled, and massively productive. We also need to develop organisational structures where Uber Programmers can thrive. Cooperating with other programmers and other parties like the customer; all working together collaboratively as a single team with a single goal: working software. Common code ownership, and using code as a design comunication tool is aligned with this view.

We need to be creating more Kent Becks.

Post Agile - In search of a new Metaphor

Agile Software Development seems to be going the same way as a number of other fads in IT, such as Object Orientation and Design by Contract. I use the word fad, because of the faddish adoption curve. Firstly there are the few original proponents with a deep understanding of the new idea, then next is the herd who lack the depth of understanding, but are eager to try the new practices, which they learn by rote, parrot fashion, and then there is the demise and ridicule as the parrot fashion, "cookie cutter" adopters falter and the original idea is discarded, baby and bath water.

One of my favorite quotes is that "the label is not the thing" from the book "Language in Thought and Action" by the psychologist S.I Hayakawa. So Agile Development is just a label, not every team that chooses to use this label is doing the "thing" the original proponents had in mind. So what to do? Should we avoid labels completely in fear that they are abused? Thats one possible answer, but labels do serve a legitimate purpose. A label is a short hand that is useful in conveying a great deal of meaning. You wouldn't want to repeat the whole Agile Manifesto every time you wanted to describe projects that share its values and principals. So the label Agile is a valid one.

The problem I think is that the Agile label conjures up an image that doesn't convey the whole thing. The image is of Gazelles leaping freely through the African Bush. Swift movement, and free spirits operating without restraint.Does this image summarise the values and principles of the Agile Manifesto? Does it summarise why people do Agile development? On one level, the answer is a partial yes. From the perspective of a programmer like Kent Beck, I suppose Agile is an apt metaphor. The freedom to hustle, move quickly and use your skills to get where you need to be fast, without impediment. But does this metaphor summarise why companies like Toyota use an Agile approach? I don't think it does. For them a more powerful metaphor is Lean. This word conjures up an image of disciplined efficiency, where everything serves a purpose and waste is eliminated. No excess baggage, no fat. But does the Lean metaphor describe the whole thing?

I wonder what would have happened if the proponents of the Agile Manifesto had decided to use the word Lean instead? I suspect that board rooms would have picked up on "the thing" more readily. As its stands Agile is often misconstrued by both Managers and Developers as a collection of technical practices, even though tools and technical practices are de-emphasised in the Agile Manifesto in favor of Leadership and sound Management.

Lets look at some of the root ideas of the Agile Movement. Lets see what Edward Deming has to say (Taken from Mary Poppendiecks Leadership presentation on InfoQ):

• Appreciate the System
• A system wide view is fundamental -never sub-optimise
  
• Manage the relationships between suppliers, producers and customers
  
• Knowledge of Variation
• Most variation is "common cause" and is inherent in the System
• Trying to eliminate this variation only makes things worse
  
• Systemic problems lie beyond the powers of the individual worker
  
• To reduce variation, provide leadership in improving the System
  
• Theory of Knowledge
• Use the Scientific Method to measure and improve Systems
  
• Psychology
  
• When it comes to people, the things that make a difference are skill, pride, expertise, confidence and cooperation.
  

To me Deming sums it up well. The noticeable addition in my mind by Deming to the "science" of Management is to recognise that traditional science plays a very small role, and an appreciation of human psychology is far more important.

Perhaps this is why Agility defies precise scientific definition. You either get it or you don't. Agility is a statement about people and how they choose to collaborate. Agile skills are human powered, and are propagated by example by Leaders who understand the importance of empowering others. When talking to people who don't get it, I think I'll be avoiding the word Agile in future. Without a shared experience to pin it to, this abused label is bound to lead to confusion. For newbies perhaps I'll try the yet untainted label "Lean" instead?

On second thoughts, I think its best to avoid labels altogether and just talk about what I do and why I do it. I'm conscious that any label will inevitably fall short, because the "thing" is all encompassing. It is "Agile", "Lean", "Adaptive", "Human Powered" and... a bunch of other stuff too! Just like human pyschology the thing is multifaceted and not easily summarised.

I guess labels are great for marketing and propoganda, but not so great at conveying non-trivial ideas.

IS ORM a Dead End?

I have never been over the moon with ORM. It solves the need to write SQL in your code, and to iterate through database results sets to form data structures, but it never really addressed the mismatch between true Objects and Data in my opinion.

Before going on to explain why, I would also like to say that ORMs also facilitate Domain Driven Design. To gain this benefit though, your application is no longer relational database centric. Instead your database becomes a pseudo object repository, storing object state and object graphs. This is fine for OO programmers but can look strange to those from a pure relational DBMS background. What DDD is really saying IMO is that OO driven application design calls for an OO database. ORM and a RDMS will do, but your DDD model is still OO not Relational.

So where does this leave us? Well hopefully acknowledging that the OO Model and the Relational Data Model are just two different models. Two different ways of modeling the world around us. Which model should we choose? Well it depends on what we are trying to model and why.

Bob Martin wrote an interesting blog post on the OO Model, ORM and the Active Record pattern. According to uncle Bob, the OO Model tries to model the world in a way that provides immunity to data changes. The idea is that data is hidden (encapsulated) inside objects, and that externally no one knows or depends on the objects data type. Instead you communicate with objects through messages. The data types can change as long as the messages and the objects answer to those messages (behavior) remains the same. This is the basic idea behind polymorphisms, which provides for immunity to implementation change, including changes to the encapsulated data type.

The relational model is very different however. Its goal is to allow you to find the data you want quickly. It does this by recognising the relationships between data types and using set based maths to "join" record sets. To do this, the relational model chooses to expose all entity data. The exact opposite of encapsulation. Exposed object (entity) data is used to join records and filter results sets during querying.


So how do we square the circle? Have a model where data types are hidden, but also where we can perform powerful queries? I don't think you can, and this is why IMO almost all ORM solutions end up exposing the underlying data (all those getters and setters). In a DDD application you learn to live with this ignoring the exposed data and augmenting the data mutation methods with true OO "business" methods that provide data encapsulating, "business domain" behaviour. But given that the data is exposed, there is nothing stopping others accessing the data themselves, breaking encapsulation and bypassing business rules. In fact if you intend to do queries you need exposed data to perform joins, filters etc.

So there is no squaring the circle, and your DDD isn't truely OO. What you have are data records that can also behave like objects, but due to the lack of encapsulation afforded to true objects and the opportunity this provides to violate OO semantics, you cannot say that your design is immune to data type changes. I believe that this is Uncle Bobs main complaint with the Active Record Pattern.

I turn the argument on its head. You are producing an application where you are interested in data types, where you want to display those data types to your users and where you want to explore relationships between data types. This is what we would call a classic database application. In these applications immunity to data type changes is an impossibility. You cannot hide data types, because data is what your user is interested in. Your user is also interested in some behaviour which express business rules, but most of those rules have to do with maintaining data integrity and enforcing relationships between data entities. Your user wants to view his data.

In such an application OO encapsulation amongst domain objects serves very little purpose. Polymorphism is only useful as a means of grouping data types with common attributes, but not as a means of grouping 'objects' with common behavior, and data encapsulation becomes meaningless. So why not forget about objects and data encapsulation and use exposed mutable data types instead? Well functional languages have been using this approach for years, an hashmap (Dictionary) with name/value pairs is a mutable data type. You can represent any data type you like by nesting hashmaps. Accepting that all data will be exposed, and that data types are likely to change is a much better fit for database applications where users want to store, navigate and query data.

Given the hashmap as the primary abstraction, where does behaviour fit in? Well there is data agnostic behaviour such as "Create", "Retrieve", "Update", and "Delete" which applies to all data types. In addition to basic CRUD is querying behaviour like "Select From", "Join" and "Where condition". These are all data type agnostic and could be provided by a framework like LINQ or Rails ActiveRecord. Then there is data type specific behaviour like "Age" which calculates the age of any data type that contains the attribute "date of birth". This would need to be provided by the application programmer and associated with a set of data types.

This is why I think that Bob got it wrong in his conclusion. Data is King in data centric applications (in contrast to behaviour being King). The Active Record pattern as implemented in Rails acknowledges this fact and treats domain entities as pseudo Objects and doesn't try to pretend that they are proper objects with encapsulation. LINQ takes the same approach too.

For database applications where set based data queries are important, then ORM has always been a misnomer in my view. What we have really been doing is data-structure relational mapping. With Rails and LINQ we are now moving into "dictionary relational mapping", which in my opinion is a more natural way to model data centric applications then "ORM".

Monopolies, Cartels and Developer Choice

This recent article on the serverside has got me thinking (yet again) about the state of the software industry as a whole. The article is about a blog entry that asks Why are Sun and Microsoft now clambering onto the Ruby bandwagon? And can Ruby be the next Visual Basic?

These questions expose some interesting assumptions: A) That there is a gap (or perhaps a gulf) in the Software Development market left by VB6 that still needs filling. B) That this gap can only be filled by a language that is backed by a major vendor like Sun or Microsoft. The last of these assumptions appear to be true when you look at the demise of Dolphin Smalltalk. Dolphin Smalltalk was an ideal OO replacement for VB6. Yet it perished. Why?

Thinking about these questions raised an interesting thought. I have long concluded that Managers/Strategist and Architects make technology purchase decisions not programmers. Managers are of course risk adverse and like to stay near to the technology herd. This makes sense given that they often know little about technology themselves. Like the saying goes, no one ever got fired for buying IBM. So vendors have learnt over the years how best to sell to Managers and decision makers. As for Programmers, well they are an easier proposition. If the Managers have bought into a given technology and the herd is clearly moving in a given direction then the programmers will clamber to get the "new" technology on to their resumes.

These cultural patterns have traditionally led to monopolies such as with IBM and then Microsoft. With the rise of Java and J2EE we have witnessed yet another phenomena, that of the cartel. In response to Microsoft's near monopoly, a group of companies decided to gang together to form a cartel. On the face of it J2EE is standards based, but until very recently it has been the cartel (Sun, IBM, Oracle, BEA etc) that have dictated the overall direction. In so doing mainstream developers are left choosing between either Java/J2EE or C#/.Net. This boils down to practically no choice at all.

So how will things be in the future? Well the cartel have split up the J2EE cake amongst themselves along database lines. But it is not just the J2EE market that is split this way. the whole serverside cake seems to be split along database lines. The thought that came to me is that Oracle and DB2 shops tend to use Java/J2EE and Microsoft SQL Server shops tend to use .Net.

So the key to sales seems to be the database. This makes sense, because from a management perspective, "the database" is the companies most important technology asset. Well actually the important asset is the corporate data not the database server, but people seldom seem to make this distinction :^). So after spending almost eight years contracting and 17 years in the industry what have I witnessed? Well the database salesman comes in and sells the managers a database. On top of that they then convince the managers of the need for middle ware. If the database is DB2 then the natural choice is Websphere, Oracle then OAS, SQLServer then .Net etc. They then convince them that it is a good thing to mandate a single technology stack for everything, because having a "corporate strategy" is important. So all teams end up using DB2 and Websphere whether the application requirements and timescales merit it or not.

Into this mix has landed open source. Some parts of the open source community are aping this traditional vendor behavior. So JBoss and Red Hat Linux see themselves as a traditional server side technology company. Others are just programmers interested in programming, and sharing tools and languages that help them become more effective at delivering. But wait a minute, how can programmer centric technology succeed if it is the managers who get to make all the decisions?

Well if the managers are themselves Programmers then this approach works very nicely. Paul Graham has been blogging on this very topic for quite some time, and the Agile community worked this one out some time ago too. Everything we do needs to be programmer centric, because at the end of the day it is the quality of the code and the productivity of programmers that counts the most. Organisations that take this approach are massively productive when compared to traditional software development organizations.

So it seems to me that I need to find work with people that realise the importance of programming and understand that they don't need Oracle (Or IBM or Microsoft) to look after their precious data. Freed from management ignorance, I'll get to choose the best tools for the job, and who knows programming could even become fun again :^). It sounds like I need to be looking for work with a startup!

Ruby, Smalltalk and Lisp Revisited

In my last post I suggested that Ruby may have some advantages over Smalltalk, especially when it comes to borrowing some of the more powerful features of Lisp. First thing to say is titling the post "Ruby versus Smalltalk" wasn't the cleverest thing to do. The responses tended to contain more heat then light. I hold up may hands and take full responsibility for this. Given the amount of FUD that Smalltalk has suffered over the years, it is hardly surprising that the Smalltalk community are protective.

I've been recently looking closely at Rubinius, a Ruby implementation that uses a Smalltalk/Squeak-like VM. Rubinius borrows many of the architectural ideas of Smalltalk/Squeak and is Ruby implemented mostly in Ruby. Eventually even the VM will be implemented in a C-like variant of Ruby called Cuby, much like how the Squeak VM is implemented in Slang.

I'm pretty taken by Rubinius. The approach taken is to implement Ruby right, and in so doing they've gone back to Smalltalk. The other thing that is very impressive is that all the Ruby libraries in Rubinius are implemented using BDD. So there are Rspec specifications for all library methods. The full Ruby library is not complete yet, but using Rspec is an excellent way to finally specify the Ruby language and in so doing allow lots of people (like myself :^)) to get involved in the Rubinous project.

So back to Lisp. The Rubinius compiler generates a simple Lisp (s-expressions) as an intermediate form on the way to byte code. There are lost of good reasons to do this, many of which are dicussed here:

http://on-ruby.blogspot.com/2007/01/will-rubinius-be-acceptable-lisp.html

Another reason not mentioned is that Lisp can become a common intermediate form allowing several source languages to interoperate. So Ruby or Smalltalk could be compiled to Lisp then Lisp to byte code. Peter Frisk has already implemented a Smalltalk to Lisp parser in Vista Smalltalk so this is possible. It also fits in nicely with the language orientated programming ideas that Martin Folwer has been touting for quite some time now.

Rubinius is gaining a lot of momentum in the Ruby community. So why is Lispyness important? I think the answer is flexibility and allowing change. Something that Alan Kay said seems relevant here. The first Smalltalk implementastions where built with Lisp I believe, and whilst Smalltalk was still within the confines of Xerox Parc the language was being experiemented with and changed all the time. Alan Kay complains that once Smalltalk became public in 1983, the language became fixed. He had hoped that Smalltalk-80 would be redundnant by now, and that it would have ben replaced by something better.

The Ruby community seem up for evolution, whilst also being a very practical bunch. The Rubinius project is cooperating with the JRuby project to specify a common Ruby across all implementations, something that I believe was a problem within the Smalltalk comunity. By considering how best to bring Lisp to Ruby they are also showing an openess to change. Matz seems open to change too, Ruby 2.0 although not revolutionary, does show a willingness to deprecate old features and break backwards compatibility, in the same spirit as pre Smalltalk-80 Smalltalks.

I still think that Ruby is closer to Lisp then Smalltalk, but that doesn't stop Smalltalk inlining Lisp or adoptng more Lisp like features if it wants to. In a sense the two languages share a common root. The difference seems to be history and the two communities. The Ruby community seem to be more willing to experiement and evolve, whilst the Smalltalk community is still holding onto Smalltalk-80 as the common Smalltalk dialect. I guess the question is whether the Ruby community can evolve together and stay as one, avoiding the fragmentation that occurred with Smalltalk.

From what I've seen on the web, there is plenty of reason for optimism. If the Rubinius/JRuby people pull it off we may end up in a situation where we don't need to choose. We can mix Ruby, Smalltalk and Lisp as and when appropriate.

Ruby versus Smalltalk

Is Ruby a lesser Smalltalk? Well I use to think so, but now, after using Ruby for a while, I'm not so sure. Well Smalltalk definitely excels when it comes to tools, but as with Java, Ruby's increasing success will mean that better Ruby tools are sure to appear soon.

The reason why I'm not so sure is the ability to create macros in Ruby, in much the same way you would with Lisp. For those that don't know, a macro function is a function that writes functions, so code that writes code. Rails uses this technique all over the place. For example the famous scaffold method is a macro. Your Controller class writes itself when it gets defined at runtime. AFAIK this just isn't possible in Smalltalk or is it? Why is this important? Because you can add new control structures to your language, extending the language and creating DSLs in much the same way you can with Lisp macros.

It looks as though Ruby is a true successor to Lisp in a way that Smalltalk never has been. Is this true or am I missing something?

Open Source Software Businesses

Following on from my last post, that was motivated by this discussion on TSS. In answer to the question. What motivates Gavin King when he is selling Seam? Well after some mis-communication, Bill Burke of JBoss was very honest about this. Jboss is a Software Business, they make their money mostly through support subscriptions. So JBoss sell support licenses for a Software Product, The JBoss Application Server. This is the business that Red Hat bought.

So is there anything wrong with this? Well no if you're open and transparent about it. I definitely do not work for free, and as Bill rightly points out, much of what JBoss as achieved in the open source arena would not have been possible if they hadn't grown the business. Also, one size doesn't fit all, so although J2EE5 isn't the simplest way to build a web application, as some of the comments to my last post rightly point out, in some cases J2EE5 is a good option.

So where does that leave us? Well I still feel that describing JBoss as "community driven" like they do on their website isn't completely "accurate" :^). JBoss like any other business will do what it sees to be in it's own interest. So putting back on my cynical hat. Gavin Kings motivation when selling Seam? Create an opportunity to sell more JBoss Subscriptions perhaps?

Again, fine - but does this qualify him as a "community leader"? In my opinion no.

The J2EE King is Dead - Long live the J2EE King!

I've quit posting on Objects. Gilad Bracha is much better qualified then I and besides he has a cool blog. Gilad reckons that Java is not OO. Well that's something coming from the guy who led development on the Java VM for a number of years.

As for me, I've decided to take Gilads advice and ensure that my next project is not a Java one. I'm seriously getting into Ruby and Rails, and I'm also dabbling with Flex. I'm finding the Ruby forums kind of boring though, basically filled of helpful knowledgeable people, supplying useful technical tit-bits, without an axe to grind insight.

In contrast the Java forums like TSS are a lot more "fun". I've noticed a trend lately, the OSS projects that exposed the swindle known as the J2EE Application Server are now seizing the J2EE crown for themselves. Spring, which is a neat framework, that basically helps to overcome a Java language smell (Gilad discusses this too), now sees itself as the new J2EE King. I have read several articles from Rod Johnson, and whilst he is very eloquent in describing the pitfalls in J2EE and in particular EJB, at no point does he mention that IoC is merely a pattern to deal wit the fact that object construction cannot be done in an OO fashion with Java. Now is this just an oversight on his part, or is he just simply ignorant of this fact?

Also, Johnson seems to have got himself into a squabble with the JBoss/Hibernate guys. The JBoss camp has an alternative vision for the new J2EE, and they see themselves as the true successors to the J2EE throne. Again, at the core is a framework to deal with the limits of the Java language. Hibernate with the use of CGLIB manages to overcome the static nature of Java, by late-binding POJOs to a database agnostic persistence mechanism at runtime. Has it escaped Gavin Kings notice that Rails manages to do the same thing, within the Ruby language and without having to resort to ugly tricks?

You would have guessed that both of these clever guys would have realised that the problem is with the language, and that they would be spending their time building OO frameworks using a more appropriate, 'OO' programming language, but they aren't, so why?

Well the simple answer is money. Java is an established market, and they both intend to exploit it. Unlike the old guard, which they have successfully over thrown, these new upstarts have an open source business model. I'm not an expert on OSS, but apparently you can make money out of software by giving it away. The thing about these new pretenders however is that in my opinion, they are both fundamentally dishonest. I don’t believe either of them is being totally open about their true motives. When an Oracle salesman walks up to you in a pin-stripped suite and offers to sell you a product, you know what you’re potentially getting into, but when Gavin King offers you Seam as the solution to all your problems what are his motives? Is he truly altruistic ?

Marc McFluery of JBoss fame is a very rich guy. So how did he make his money? The JBoss camp has chosen to stick close to the JCP and build on Java "standards". Their latest offering in this vain is Seam 2.0, which builds on JSF1.? and EJB3.0. As I understand it JBoss with Seam is fully J2EE 5 compliant. So I guess this is what Red Hat thought they bought. McFluery must of laughed all the way to the bank!

So back to Gavin and his Seam2.0. Without going into details, Seam tries to do with J2EE5 what Rails does with Ruby. Now I guess you're saying that a full blown J2EE5 App Server is not needed to do what Rails does, even with a limited language like Java - yes, true, but... JBoss is J2EE5 standards complaint so they must use all the J2EE5 standards :^).

From all accounts Seam at its core is an elegant framework, the problem is though is that it solves the wrong problem. The problem it should be solving is how do I build web applications with Java quickly? (to which the obvious answer is, you don't :^)). Instead the problem it chooses to solve is how do I build web applications using J2EE5, including EJB3.0, JSF and a full blown App Server? If you sell Application Servers, or product consulting for Application Servers like JBoss does, then perhaps this latter problem does need a solution. So back to the question, what are Gavin Kings motives when he is selling Seam? Well if you are as cynical as me, you would say that it is to push JBoss as an Application Server and to sell more product consulting.

These political and business issues have dogged the IT industry for a very long time, yet they are never adequately debated amongst the developer community in my view. My concern is that there will be a lot of young naive developers out there that believe that they can trust the likes of Rod Johnson and Gavin King as honest “community” leaders, and will actually invest hours learning either Spring or Seam, only to find out that it all was one big con and that their time would have been much better spent learning Ruby.

Anyway, like I say I intend to be doing less Java in the future. Personally, I prefer not to identify with a specific programming language and I like to see myself as just a Programmer first and foremost; that way I’m free to choose any programming language that suites me. My advice to the Java community though, if they are listening, is to watch out for the new pretenders. They could turn out to be worst then the ‘leaders’ we have now.