In my last post there was a certain amount of “Son, I’ve seen more large projects, than you’ve had hot dinners” attitude. So in this blog post I offer my thoughts on dealing with complexity and large projects.
Defining complexity
The problem with complexity is that it resists easy definition, but you know it when you see it, or more likely you know it when you’re in a middle of massive deathmarch project with no end in sight. However, for the purposes of this post I shall define it along two axes: time and chimera component.
- Time – Any project or task you think will take your team or organisation more than 3 months to complete can be considered complex.
- Chimera Component – Like the many-headed beast of legend, the greater number of features/requirements/scope – the more complex the project. Keep adding features (or heads) and the project will eventually eat you.
Of course it’s the chimera component that is the true test of complexity. The building industry implicitly understands that it’s a bad idea to build a house that is also a shopping mall, or an office block that is also a bridge. Likewise the car industry realises it’s a bad idea to create a car that’s also a boat, and if you do create the car-bo, it’ll be expensive and be a rubbish car and a rubbish boat.
Yet in the software industry we create these types of chimeras all the time. The #1 offender for signing off chimera projects is the government, who you would imagine, after years of IT fiascos NHS, ID cards etc, would think – maybe we should reign the scope in a bit. According to the Independent the British taxpayer has squandered £26Bn on failed IT projects since 2000 (and to think we’re worried about banker bonueses).
Big business runs government a close second, but the failures are less widely publicised. In fact a recent Computer Weekly article says that large IT projects are 20 TIMES more likely to end in catastrophe than any other type of project. All disgraceful and embarrassing statistics for all of us working in the industry.
Even a stopped clock tells the right time twice a day
However, from time to time we manage to deliver a success. For example, I am in awe of 
London’s Oyster Card system which was largely delivered on time and on budget, contains 1.5 million lines of code, and successfully deals with over 7 million commuter journeys per day.
So what made Oyster a success and NHS a disaster? I’m not pretending I’ve worked on or managed one of these ultra projects, but here’s my views on pragmatic steps you can take to try to avoid calamity.
1 – Turnover is vanity, profit is sanity
First a quick anecdote. Back in 2002 the company I was working for was reeling from the dotcom bubble bursting and the silver haired boss of the American parent company was dispatched to layoff some of us minions. He gave the usual speech with the usual BS about how this was more painful for him than us etc. However, he made one point that has stuck with me to this day and was one that I hadn’t really considered before
All businesses exist to turn a profit
It looks obvious when written down, but while I was worrying about the minutiae of my Delphi project (those weren’t the days), I hadn’t thought of the wider context of – has this project got any chance of turning a profit. Given the layoffs – the answer was clearly no.
However, many working in IT act as if they have all the time in the world to work on vanity projects, specious methodologies and dubious features without any understanding that they are based in the real world of capitalism and profit & loss.
So first analyse your project through the lens of profitability. Does the timescale, and price for the project look realistic, and if not cull those features that appear less useful to the user and shorten the delivery and scope.
2 – Phased delivery of working USEFUL software
Agile methodologies all have phased delivery built in. At the end of each iteration you must deliver working software. However, working software is not necessarily useful software. If you have kicked off multiple features simultaneously you may deliver 10 features that are all 5% complete and are in actual fact no use to anybody.
So using step 1 you’ve hopefully identified the features you think are profitable ones, now only start work on one or two of those so you should get the team to deliver it at more like 60% completeness. So the user can assess it straight away, feedback on it etc. You should then endeavour to get that feature to 100% completeness in the next iteration.
3 – Three month mini projects
Split the project into a series of 3 month mini-projects, each 3 month deliverable should aim to deliver part of the project but containing only complete features. The customer can see progress and potentially has a walkaway point, or can reprioritise featues based on progress.
Important – Assess critically if the project still makes sense, is still on track to make a profit. Assess if the features sitting at the top of the backlog are still high priority. Has the tech changed, has the environment changed, is there still buy-in from the customer etc. Has the market changed, eg Ipad platform now a priority – so ditch Flash etc. Take action and kill features or parts of the project that no-longer make sense or threaten to derail the project.
Item: “Large projects nearly always fail unless broken into components that can be delivered step by step”
4 -YAGNI and DTSTTCPW must be your religion
YAGNI and DTSTTCPW are a much better guide than the latest fads within agile.
Everyone says it, but you must keep the team tightly disciplined on scope creep. It’s so very easy to slip into Gold-Plating features that don’t need it and worrying about optimisation early. It’s surprising the number of projects I’ve been on that have a massive Oracle DB backend and are probably only doing a few 1000 transactions per day As Eric Ries says deliver the minimum viable product.
Item: “process has greater emphasis than outcome and that’s not going to get a project over the line”
Conclusion
So while we can all agree that small is beautiful when it comes to IT projects the rub is that if you want to make some serious money, you’ll inevitably have to sign-up to some form of complex project. Ie a 3month + project with a large chimera component.
However, if you manage complexity with an eye towards profitablility, delivering working features (not just working software) that have survived the Occam’s razor of YAGNI and DTSTTCPW and you aim to have an actual product every 3 months you can do a lot to mitigate complexity.
Of course we’re our own worst enemies. If we’re asked to tender for the next NHS system, instead of questioning the scope, we’ll put in a low bid and hope the government is stupid enough to sign it off. When we walk away £10bn richer for a failed delivery, we know we won’t be punished for our failures. Equally people putting £10bn contracts out to tender should by now know a lot better and stop putting such massive scope into them. But that’s another story…
I do feel agile (applied without all the ceremony and fluff) is the best solution we’ve got to delivering working software into the hands of end-users quickly and to a high-degree of quality. We can cut the heads off the chimera one by one, and win.
Convivial Coding 