Agile – Dealing with complexity by avoiding chimeras

In my last post there was a certain amount of “Son, I’ve seen more large projects, than you’ve had hot dinners” attitude.  So in this blog post I offer my thoughts on dealing with complexity and large projects.

Defining complexity

The problem with complexity is that it resists easy definition, but you know it when you see it, or more likely you know it when you’re in a middle of massive deathmarch project with no end in sight.  However, for the purposes of this post I shall define it along two axes: time and chimera component.

  • Time –  Any project or task you think will take your team or organisation more than 3 months to complete can be considered complex.
  • Chimera Component – Like the many-headed beast of legend, the greater number of features/requirements/scope – the more complex the project.  Keep adding features (or heads) and the project will eventually eat you.

Of course it’s the chimera component that is the true test of complexity.  The building  industry implicitly understands that it’s a bad idea to build a house that is also a shopping mall, or an office block that is also a bridge.  Likewise the car industry realises it’s a bad idea to create a car that’s also a boat, and if you do create the car-bo, it’ll be expensive and be a rubbish car and a rubbish boat.

Yet in the software industry we create these types of chimeras all the time.  The #1 offender for signing off chimera projects is the government, who you would imagine, after years of IT fiascos NHS, ID cards etc, would think – maybe we should reign the scope in a bit.  According to the Independent the British taxpayer has squandered £26Bn on failed IT projects since 2000 (and to think we’re worried about banker bonueses).

Big business runs government a close second, but the failures are less widely publicised.  In fact a recent Computer Weekly article says that large IT projects are 20 TIMES more likely to end in catastrophe than any other type of project.  All disgraceful and embarrassing statistics for all of us working in the industry.

Even a stopped clock tells the right time twice a day

However, from time to time we manage to deliver a success.  For example, I am in awe of London’s Oyster Card system which was largely delivered on time and on budget, contains 1.5 million lines of code, and successfully deals with over 7 million commuter journeys per day.

So what made Oyster a success and NHS a disaster?  I’m not pretending I’ve worked on or managed one of these ultra projects, but here’s my views on pragmatic steps you can take to try to avoid calamity.

1 – Turnover is vanity, profit is sanity

First a quick anecdote.  Back in 2002 the company I was working for was reeling from the dotcom bubble bursting and the silver haired boss of the American parent company was dispatched to layoff some of us minions.  He gave the usual speech with the usual BS about how this was more painful for him than us etc.  However, he made one point that has stuck with me to this day and was one that I hadn’t really considered before

All businesses exist to turn a profit

It looks obvious when written down, but while I was worrying about the minutiae of my Delphi project (those weren’t the days), I hadn’t thought of the wider context of – has this project got any chance of turning a profit.  Given the layoffs – the answer was clearly no.

However, many working in IT act as if they have all the time in the world to work on vanity projects, specious methodologies and dubious features without any understanding that they are based in the real world of capitalism and profit & loss.

So first analyse your project through the lens of profitability.  Does the timescale, and price for the project look realistic, and if not cull those features that appear less useful to the user and shorten the delivery and scope.

Item: “[NHS System] intended as a complete “big-bang” replacement for the many and varied existing EHR systems”

2 – Phased delivery of working USEFUL software

Agile methodologies all have phased delivery built  in.  At the end of each iteration you must deliver working software.   However, working software is not necessarily useful software.  If you have kicked off multiple features simultaneously you may deliver 10 features that are all 5% complete and are in actual fact no use to anybody.

So using step 1 you’ve hopefully identified the features you think are profitable ones, now only start work on one or two of those so you should get the team to deliver it at more like 60% completeness.  So the user can assess it straight away, feedback on it etc.  You should then endeavour to get that feature to 100% completeness in the next iteration.

Item: ” a crucial decision [in Oyster’s success] was to revise the original plan of a “big bang” introduction in favour of a phased roll-out”

3 – Three month mini projects

Split the project into a series of 3 month mini-projects, each 3 month deliverable should aim to deliver part of the project but containing only complete features.  The customer can see progress and potentially has a walkaway point, or can reprioritise featues based on progress.

Important – Assess critically if the project still makes sense, is still on track to make a profit.  Assess if the features sitting at the top of the backlog are still high priority.  Has the tech changed, has the environment changed, is there still buy-in from the customer etc.  Has the market changed, eg Ipad platform now a priority – so ditch Flash etc.  Take action and kill features or parts of the project that no-longer make sense or threaten to derail the project.

Item: “Large projects nearly always fail unless broken into components that can be delivered step by step”

4 -YAGNI and DTSTTCPW must be your religion

YAGNI and DTSTTCPW are a much better guide than the latest fads within agile.

Everyone says it, but you must keep the team tightly disciplined on scope creep.  It’s so very easy to slip into Gold-Plating features that don’t need it and worrying about optimisation early.  It’s surprising the number of projects I’ve been on that have a massive Oracle DB backend and are probably only doing a few 1000 transactions per day  As Eric Ries says deliver the minimum viable product.

Item: “process has greater emphasis than outcome and that’s not going to get a project over the line”


So while we can all agree that small is beautiful when it comes to IT projects the rub is that if you want to make some serious money, you’ll inevitably have to sign-up to some form of complex project.  Ie a 3month + project with a large chimera component.

However, if you manage complexity with an eye towards profitablility, delivering working features (not just working software) that have survived the Occam’s razor of YAGNI and DTSTTCPW and you aim to have an actual product every 3 months you can do a lot to mitigate complexity.

Of course we’re our own worst enemies.  If we’re asked to tender for the next NHS system, instead of questioning the scope, we’ll put in a low bid and hope the government is stupid enough to sign it off.  When we walk away £10bn richer for a failed delivery, we know we won’t be punished for our failures.  Equally people putting £10bn contracts out to tender should by now know a lot better and stop putting such massive scope into them.  But that’s another story…

I do feel agile (applied without all the ceremony and fluff) is the best solution we’ve got to delivering working software into the hands of end-users quickly and to a high-degree of quality.  We can cut the heads off the chimera one by one, and win.

Agile punks – Go write an app

Last week I ranted posted about how some members of the agile community are engaging in pointless debates about complexity theory, kanban maestros & scrum blockades. I pointed to a post by Liz Keogh as an archetype of the genre (as I said to Liz, there’s plenty of other authors I could have used).  Liz asked for some feedback and was generous enough to agree that I had a point.

If we try to ignore the first commenter, who hilariously and spectacularly, misses the point by wondering if the issue is a “complex problem, possibly even chaotic”.  I think Liz has drawn all the wrong conclusions, about the state of the industry.

It is the best of times, it is the worst of times

In my 14 years in the industry I’m confident that there has never been a better time to be a programmer or to work in IT.  But thanks to a community of well intentioned “agilists” many in the industry are flagellating themselves into believing themselves hopeless programmers.

I live in Scotland and there’s an old joke about the definition of a Scottish Calvinist:

“Someone who is paralysed by the fear that someone, somewhere, is enjoying themselves”

Similarly the definition of a modern programmer could be:

“Someone who is paralysed by the fear that someone, somewhere, is doing BDD better than they are”

Many programmers are so obsessed with writing better unit tests, understanding the problem better, worrying about their kanban wall, working through Martin Fowlers pattern library, trying to be “masters” not “journeymen” that they’ve forgotten the basic joy of cranking out some code and solving a users problem, and are paralysed by the fear that they’re doing it wrong or poorly.

Ironically, it’s never been easier to write some code and get it downloaded and used.  Individual devs can create and upload apps to the App-store for a potential audience of millions.  Cloud computing means the home coder can create apps that can scale massively on demand for an initial outlay of a less than £20 quid.  While you’ve been worrying about your BDD adverbs the iFart dev just made $40,000.

It’s not that improving your skills isn’t important, it just shouldn’t become all consuming and derail shipping software.  Apart from my blog posts, there is no such thing as perfection, and guess what, you learn from your mistakes. And we certainly shouldn’t be putting up barriers to welcome all and sundry into the industry, it sure beats working at Tesco.  If you are a “master” your one mission in life should be to make it easier not harder for “journeymen” to improve (BTW I loathe this master/journeyman patronising nonsense, it just succeeds in making programmers more paranoid).

Are you as punk as Ward?

I like to think of Ward Cunningham, Kent Beck, et al as the first punks in our industry.  Just like the original punk rock was a reaction to the complex and multi-instrumented prog music as epitomised by bands like Yes and Emerson Lake and Palmer.  XP was a reaction against the “analysis paralysis” caused by, as they saw it, the complex and formal software development methodologies of the day.  By turning things like TDD, Small releases, CI up to 11 they managed to break the deadlock and start rocking out the code.

XP was the forefather to agile, and sadly since those heady simple days of XP, we’ve slowly been adding the complexity back in – until I overheard someone say recently “I’m not sure I understand the definition of a story, properly” as if that were important.  We’re now in a state of “over-analysis paralysis”.

Anarchy in the IDE

I’m not the only one thinking like this, Jeff Atwood recently tweeted:

“all this process stuff at #ordev is boring. At Stack Exchange, we have one process: kicking ass and chewing gum. And we’re all out of gum.”

Zed Shaw sends up the agile community far better than I can using a tad more colourful language at Programming Motherf***ker

So I propose we need to bring back some of that original punk rock spirit.  There was a famous punk t-shirt which said “Here’s 3 chords, now form a band”

Similarly in programming we could do with something similar:

  1. Here’s an IF statement
  2. Here’s a loop
  3. Here’s an editor
  4. Now write an app

Or as Johnny Rotten said “Don’t accept the old order. Get rid of it.”