Achieving Consensus – It’s so funny, how we don’t talk anymore

Walk into any office and there can usually be found a surplus of opinion and a dearth of consensus and agreement. While the crowd may be wise, as individuals we all know we’re right and damn any fool who disagrees with us.

A quick perusal of the comments section of any, even moderately controversial post, and you’ll see an echo chamber of hundreds of people carefully explaining why each other’s opinion is completely and hopelessly wrong.

The rarest phrase on the internet, is – “That’s a compelling argument, you’ve completely changed my mind”

In fact, the comments sections of most blogs are now so poisonous that the creators of the content, and the majority of its readers, don’t even bother to “Read below the line”.  Worse, people with interesting and unusual opinions are intimidated into not broadcasting them in the first place.

I’d like to say we can all agree that this is a terrible state of affairs, but as I’ve explained, that’s merely an invitation to be pointed at 100 other blog posts explaining why I’m wrong.

So all we can say is that opinions are cheap, and getting consensus is the hard part.  For a contemporary example – the recent Paris climate change talks are both simultaneously The World’s Greatest Diplomatic Achievement, and A Squalid Retrenchment.

Picking battles and getting permission

I wanted to write this post, not because I have the answer to generating constructive discussion on the internet (I wish).  But because I think teams and individuals (including myself) struggle on the projects we work on every day – not for want of ability but for want of consensus and clarity on what we want to achieve.

Because we all instinctively understand the difficulty and energy required to get a group of people to agree on just about anything – the path of least resistance is to avoid the discussion in the first place. Indeed by having the discussion you expose yourself to the risk that your argument will not carry the day, and you’ll be forced down a road you’d prefer not to be on. So it’s easy to understand why tough decisions and “honest” conversations are avoided and “difficult” people worked around.

The old cliché “Pick your battles” or my personal favourite “Better to ask forgiveness than get permission” are undoubted truisms, but they come with a cost.  By avoiding the battles and not getting the permission, in other words not getting consensus, you can find yourself isolated when you hit the inevitable bumps in the road.  As you’ve neatly provided those excluded from the decision with a convenient scape-goat.  eg – “Well if he’d asked me, of course I’d have told him that was a terrible idea.”

Or as Sun-Tzu put it more pithily:

“Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win” 

Are you thinking what I’m thinking?

I’m certainly not advocating that every decision needs to be ran past the team, but decisions that will impact more than one or two people or that will have a significant impact on what is delivered,  and when it will be delivered, should be discussed as a group.

Keeping the team and the business involved in the decision making process will cost you time and you’ll inevitably have to slaughter some of your personal sacred cows, to reach an approach everyone can agree on.  However, the prize is worth the pain.

That prize is a clearly articulated plan that both the team and business believes in and much more importantly – are invested in.  Reason being – because you’ve arrived at the plan as a group, the group should want it to succeed as they all have some some skin in the game.

The drive and pressure to “Get stuff done” and the small windows many agile methodologies allow for meetings and planning between sprints means it’s easy to skip getting the team bought into the plan and approach. This is a mistake.

Rather than thinking of getting consensus as an exercise in cost/pain/stress think of it as an extremely valuable deliverable and vital part of leading a team (unfortunately the cost/pain/stress are still there 😉

In the end it’s just all about talking, and remember to heed Cliff’s warning…

Advertisement

Visual Studio Turbo – DIY AppHarbor with Nant.Builder

In the final part of this series I look at automating uploading your app into the Windows Azure Cloud, or as I like to think of it a Do It Yourself AppHarbor, hopefully with no leftover screws ;-).  The series so for:

1. Visual Studio 2010 Workflow
2. Automating Your Builds with Nant.Builder
3. DIY AppHarbor – Deploying Your Builds onto Windows Azure

Update 08/08/12 – Updated Nant.Builder and links to reflect changes for Azure 1.7 and Azure Powershell Commandlets

Prerequisites

1.  You’ll hopefully not be surprised to learn you’re going to need a Windows Azure account (there’s a rather stingy 90 day free trial, if you haven’t signed up already).  Within your account you’re going to need to set up one Hosted Service where we’ll deploy the app to, and one Storage Account where the package gets uploaded to prior to deployment.  If you’re stuggling just Google for help on configuration and setting up Windows Azure, there’s plenty of good guides out there.

2. You’ll also need to install the .net Windows Azure SDK v1.7.  Again I’ll assume you know how to add and configure an Azure project to your solution.

3.  Finally, you need to download the Windows Azure Powershell Cmdlets.  This will be installed automatically using Web Platform Installer.  Follow the Getting Started instructions here to ensure it was successfully installed.  You can get a list of available commands, here.

Getting Started – Importing your Azure Credentials

• You’re going to need to download your Azure credentials, so Nant.Builder can contact Azure on your behalf.  We can do this by clicking here:
  • https://windows.azure.com/download/publishprofile.aspx?wa=wsignin1.0
• You should now have file called <your-sub>-<date>-credentials.publishsettings
  • Unhelpfully you can’t seem to rename the file on the portal to make it more meaningful
• If you open the file you’ll see it’s an XML file containing your subscription details.
  • IMPORTANT– if you have multiple azure subscriptions you’ll need to edit the file so that it only includes the one subscription that you want to deploy your app into.
• With the file downloaded open powershell and run the following commands, note you’ll need to change the path and filename to your .publishsettings file:

Import-AzurePublishSettingsFile –PublishSettingsFile 'c:\users\<username>\downloads\your-credentials.publishsettings' –SubscriptionDataFile 'c:\dev\tools\windowsazure\subscriptions\your-sub.xml'

view raw

ImportAzureSettings.ps1

hosted with ❤ by GitHub

• If the above command run successfully you should have an xml containing your subscriptionId and thumbprint in c:\dev\tools\windowsazure\subscriptions
• ** REALLY IMPORTANT** – The subscription xml file is basically the keys to your Azure account, so you DO NOT want to be casually emailing it around, take it to the pub etc.  Ensure you save it behind a firewall etc etc.
• OK that’s us got our Azure credentials organised, next we can configure Nant.Builder

Configure Nant.Builder for Azure Deployment

Packaging your solution for Azure

• Install and configure Nant.Builder as described in Part 2 of this series.
• Open the Nant.build file and navigate to the Azure Settings section.
• Set the create.azure.package parameter to true, this will call CSPack to package your solution in a format suitable for deployment to Windows Azure.  If you’re interested in what’s happening here I’ve talked about CSPack in depth here and here
• Set the azure.project.name parameter to the name of the Azure project in your solution.
• Set the azure.role.project.name parameter to the name of the project which contains the entrypoint to your app.  This will most likely be the Web project containing your MVC views etc.
• Finally set the azure.service.config.file parameter to the name of the *.cscfg file containing the Azure config you want to deploy.  The default is *.cloud.cscfg but may be different if you have a test config, live config etc.
• You can run Nant.Builder now and your solution should be packaged and output in C:\dev\releases\<your-solution-name>

Deploying your solution to Azure

• If packaging has succeeded, you can now finally automate deployment to Azure.  Navigate to the Azure deployment section within Nant.Build
• Set the deploy.azure.package parameter to true
• Set the azure.subscription.credentials.file parameter to the name of the the file you created in the Import your Azure Credentials section above, ie C:\dev\tools\WindowsAzure\Subscriptions\yourSubscription.xml
• Set the azure.hosted.service.name parameter to the name of the hosted service you want to deploy your app into.  IMPORTANT – be aware that this is the name listed as the DNS Prefix not the actual service name

• Set the azure.deployment.environment parameter to the environment type you wish to deploy your app into.  Valid values are either staging or production
• Finally set the azure.storage.account.name parameter to the name of the storage account you set up earlier, this is where the app will be uploaded to temporarily when it’s being deployed.
• That’s it we should now be ready to test our DIY App Harbor.  Your Azure Config section should look similar to this, obviously with your app details replaced:

 <!--Azure Settings-->

<!-- Packaging -->
 
 <!--The name of the project containing the Azure csdef, cscfg files-->
 
 <!-- This is the name of the project containing your app entry point, probably the Web project, but may be a library if using a worker role-->
 
 <!-- The name of the file containing the azure config for your app, default is .Cloud but may be custom if you have multiple configs, eg test, live etc -->


<!-- Deployment -->
 
 <!-- The name of the file containing your exported subcription details - IMPORTANT keep this file safe as it contains very sensitive credentials about your Azure sub -->
 
 <!-- The name of a azure hosted service where you want to deploy your app-->
 
 <!-- The environment type either Staging or Production-->
 
 <!-- The name of a storage account that exists on your subscription, this will be used to temporarily load your app into while it's being deploed-->
 

One Click Deployment

So we have hopefully achieved the dream of all modern developers being able to deploy our app into the cloud with one click.  If it’s successful you should see something similar to

DeployAzurePackage:

     [exec] 27/05/2012 22:54 - Azure Cloud App deploy script started.
     [exec] 27/05/2012 22:54 - Preparing deployment of ContinuousDeploy to your service
     [exec] or inception with Subscription ID your subid
     [exec] 27/05/2012 22:54 - Creating New Deployment: In progress
     [exec] 27/05/2012 22:56 - Creating New Deployment: Succeeded, Deployment ID
     [exec] 27/05/2012 22:56 - Starting Instances: In progress
     [exec] 27/05/2012 22:56 - Starting Instance 'Nant.Builder.Sample.Web_IN_0': Stopped
     [exec] 27/05/2012 22:57 - Starting Instance 'Nant.Builder.Sample.Web_IN_0': Initializing
     [exec] 27/05/2012 23:00 - Starting Instance 'Nant.Builder.Sample.Web_IN_0': Busy
     [exec] 27/05/2012 23:01 - Starting Instance 'Nant.Builder.Sample.Web_IN_0': Ready
     [exec] 27/05/2012 23:01 - Starting Instances: Succeeded
     [exec] 27/05/2012 23:01 - Created Cloud App with URL http://xxx
     [exec] 27/05/2012 23:01 - Azure Cloud App deploy script finished.

BUILD SUCCEEDED

Note – You are better to run Nant from the command line to see the above output, as the powershell script that deploys your build echos progress to the command line, but not to Visual Studio, if you are running Nant as an external tool

Nant.Builder.Sample

I’ve created a sample project on GitHub that shows Nunit.Builder integrated into it, so it should be more obvious how it all wires up.  Download Nant.Builder.Sample here

Conclusions

I hope you’ve found the series useful, and that you benefit from turbo-charging your workflow.  Over the next month I’m going to refactor Nant.Builder to be a bit more modular, so it will be easy for other to extend the platform with different targets.  Stay tuned for further exiting announcements 🙂

Agile – Dealing with complexity by avoiding chimeras

In my last post there was a certain amount of “Son, I’ve seen more large projects, than you’ve had hot dinners” attitude.  So in this blog post I offer my thoughts on dealing with complexity and large projects.

Defining complexity

The problem with complexity is that it resists easy definition, but you know it when you see it, or more likely you know it when you’re in a middle of massive deathmarch project with no end in sight.  However, for the purposes of this post I shall define it along two axes: time and chimera component.

• Time –  Any project or task you think will take your team or organisation more than 3 months to complete can be considered complex.
• Chimera Component – Like the many-headed beast of legend, the greater number of features/requirements/scope – the more complex the project.  Keep adding features (or heads) and the project will eventually eat you.

Of course it’s the chimera component that is the true test of complexity.  The building  industry implicitly understands that it’s a bad idea to build a house that is also a shopping mall, or an office block that is also a bridge.  Likewise the car industry realises it’s a bad idea to create a car that’s also a boat, and if you do create the car-bo, it’ll be expensive and be a rubbish car and a rubbish boat.

Yet in the software industry we create these types of chimeras all the time.  The #1 offender for signing off chimera projects is the government, who you would imagine, after years of IT fiascos NHS, ID cards etc, would think – maybe we should reign the scope in a bit.  According to the Independent the British taxpayer has squandered £26Bn on failed IT projects since 2000 (and to think we’re worried about banker bonueses).

Big business runs government a close second, but the failures are less widely publicised.  In fact a recent Computer Weekly article says that large IT projects are 20 TIMES more likely to end in catastrophe than any other type of project.  All disgraceful and embarrassing statistics for all of us working in the industry.

Even a stopped clock tells the right time twice a day

However, from time to time we manage to deliver a success.  For example, I am in awe of London’s Oyster Card system which was largely delivered on time and on budget, contains 1.5 million lines of code, and successfully deals with over 7 million commuter journeys per day.

So what made Oyster a success and NHS a disaster?  I’m not pretending I’ve worked on or managed one of these ultra projects, but here’s my views on pragmatic steps you can take to try to avoid calamity.

1 – Turnover is vanity, profit is sanity

First a quick anecdote.  Back in 2002 the company I was working for was reeling from the dotcom bubble bursting and the silver haired boss of the American parent company was dispatched to layoff some of us minions.  He gave the usual speech with the usual BS about how this was more painful for him than us etc.  However, he made one point that has stuck with me to this day and was one that I hadn’t really considered before

All businesses exist to turn a profit

It looks obvious when written down, but while I was worrying about the minutiae of my Delphi project (those weren’t the days), I hadn’t thought of the wider context of – has this project got any chance of turning a profit.  Given the layoffs – the answer was clearly no.

However, many working in IT act as if they have all the time in the world to work on vanity projects, specious methodologies and dubious features without any understanding that they are based in the real world of capitalism and profit & loss.

So first analyse your project through the lens of profitability.  Does the timescale, and price for the project look realistic, and if not cull those features that appear less useful to the user and shorten the delivery and scope.

Item: “[NHS System] intended as a complete “big-bang” replacement for the many and varied existing EHR systems”

2 – Phased delivery of working USEFUL software

Agile methodologies all have phased delivery built  in.  At the end of each iteration you must deliver working software.   However, working software is not necessarily useful software.  If you have kicked off multiple features simultaneously you may deliver 10 features that are all 5% complete and are in actual fact no use to anybody.

So using step 1 you’ve hopefully identified the features you think are profitable ones, now only start work on one or two of those so you should get the team to deliver it at more like 60% completeness.  So the user can assess it straight away, feedback on it etc.  You should then endeavour to get that feature to 100% completeness in the next iteration.

Item: ” a crucial decision [in Oyster’s success] was to revise the original plan of a “big bang” introduction in favour of a phased roll-out”

3 – Three month mini projects

Split the project into a series of 3 month mini-projects, each 3 month deliverable should aim to deliver part of the project but containing only complete features.  The customer can see progress and potentially has a walkaway point, or can reprioritise featues based on progress.

Important – Assess critically if the project still makes sense, is still on track to make a profit.  Assess if the features sitting at the top of the backlog are still high priority.  Has the tech changed, has the environment changed, is there still buy-in from the customer etc.  Has the market changed, eg Ipad platform now a priority – so ditch Flash etc.  Take action and kill features or parts of the project that no-longer make sense or threaten to derail the project.

Item: “Large projects nearly always fail unless broken into components that can be delivered step by step”

4 -YAGNI and DTSTTCPW must be your religion

YAGNI and DTSTTCPW are a much better guide than the latest fads within agile.

Everyone says it, but you must keep the team tightly disciplined on scope creep.  It’s so very easy to slip into Gold-Plating features that don’t need it and worrying about optimisation early.  It’s surprising the number of projects I’ve been on that have a massive Oracle DB backend and are probably only doing a few 1000 transactions per day  As Eric Ries says deliver the minimum viable product.

Item: “process has greater emphasis than outcome and that’s not going to get a project over the line”

Conclusion

So while we can all agree that small is beautiful when it comes to IT projects the rub is that if you want to make some serious money, you’ll inevitably have to sign-up to some form of complex project.  Ie a 3month + project with a large chimera component.

However, if you manage complexity with an eye towards profitablility, delivering working features (not just working software) that have survived the Occam’s razor of YAGNI and DTSTTCPW and you aim to have an actual product every 3 months you can do a lot to mitigate complexity.

Of course we’re our own worst enemies.  If we’re asked to tender for the next NHS system, instead of questioning the scope, we’ll put in a low bid and hope the government is stupid enough to sign it off.  When we walk away £10bn richer for a failed delivery, we know we won’t be punished for our failures.  Equally people putting £10bn contracts out to tender should by now know a lot better and stop putting such massive scope into them.  But that’s another story…

I do feel agile (applied without all the ceremony and fluff) is the best solution we’ve got to delivering working software into the hands of end-users quickly and to a high-degree of quality.  We can cut the heads off the chimera one by one, and win.

Agile punks – Go write an app

Last week I ranted posted about how some members of the agile community are engaging in pointless debates about complexity theory, kanban maestros & scrum blockades. I pointed to a post by Liz Keogh as an archetype of the genre (as I said to Liz, there’s plenty of other authors I could have used).  Liz asked for some feedback and was generous enough to agree that I had a point.

If we try to ignore the first commenter, who hilariously and spectacularly, misses the point by wondering if the issue is a “complex problem, possibly even chaotic”.  I think Liz has drawn all the wrong conclusions, about the state of the industry.

It is the best of times, it is the worst of times

In my 14 years in the industry I’m confident that there has never been a better time to be a programmer or to work in IT.  But thanks to a community of well intentioned “agilists” many in the industry are flagellating themselves into believing themselves hopeless programmers.

I live in Scotland and there’s an old joke about the definition of a Scottish Calvinist:

“Someone who is paralysed by the fear that someone, somewhere, is enjoying themselves”

Similarly the definition of a modern programmer could be:

“Someone who is paralysed by the fear that someone, somewhere, is doing BDD better than they are”

Many programmers are so obsessed with writing better unit tests, understanding the problem better, worrying about their kanban wall, working through Martin Fowlers pattern library, trying to be “masters” not “journeymen” that they’ve forgotten the basic joy of cranking out some code and solving a users problem, and are paralysed by the fear that they’re doing it wrong or poorly.

Ironically, it’s never been easier to write some code and get it downloaded and used.  Individual devs can create and upload apps to the App-store for a potential audience of millions.  Cloud computing means the home coder can create apps that can scale massively on demand for an initial outlay of a less than £20 quid.  While you’ve been worrying about your BDD adverbs the iFart dev just made $40,000.

It’s not that improving your skills isn’t important, it just shouldn’t become all consuming and derail shipping software.  Apart from my blog posts, there is no such thing as perfection, and guess what, you learn from your mistakes. And we certainly shouldn’t be putting up barriers to welcome all and sundry into the industry, it sure beats working at Tesco.  If you are a “master” your one mission in life should be to make it easier not harder for “journeymen” to improve (BTW I loathe this master/journeyman patronising nonsense, it just succeeds in making programmers more paranoid).

Are you as punk as Ward?

I like to think of Ward Cunningham, Kent Beck, et al as the first punks in our industry.  Just like the original punk rock was a reaction to the complex and multi-instrumented prog music as epitomised by bands like Yes and Emerson Lake and Palmer.  XP was a reaction against the “analysis paralysis” caused by, as they saw it, the complex and formal software development methodologies of the day.  By turning things like TDD, Small releases, CI up to 11 they managed to break the deadlock and start rocking out the code.

XP was the forefather to agile, and sadly since those heady simple days of XP, we’ve slowly been adding the complexity back in – until I overheard someone say recently “I’m not sure I understand the definition of a story, properly” as if that were important.  We’re now in a state of “over-analysis paralysis”.

Anarchy in the IDE

I’m not the only one thinking like this, Jeff Atwood recently tweeted:

“all this process stuff at #ordev is boring. At Stack Exchange, we have one process: kicking ass and chewing gum. And we’re all out of gum.”

Zed Shaw sends up the agile community far better than I can using a tad more colourful language at Programming Motherf***ker

So I propose we need to bring back some of that original punk rock spirit.  There was a famous punk t-shirt which said “Here’s 3 chords, now form a band”

Similarly in programming we could do with something similar:

1. Here’s an IF statement
2. Here’s a loop
3. Here’s an editor
4. Now write an app

Or as Johnny Rotten said “Don’t accept the old order. Get rid of it.”

Agile Manifesto – The First Amendment

Riddle me this:

Q – What’s the #1 difference that an agile approach gives you versus old-skool software development methodologies (hint clue’s in the name)

A – Quickly or “agilely” getting working code into your end-users hands

It’s this answer that an awful lot of people miss, and it’s why I propose the first amendment to the Agile Manifesto:

“Any agile process should strive to get working software into the hands of end-users as quickly as possible.”

Let me now explain my reasoning:

There is an awful lot of guff talked about agile, for example, I’ve held my head in my hands after reading some recent blog posts discussing Cynefin and complexity thinking.

This sort of pontificating rubbish is what gives the consulting industry a bad name.  The main elements of developing software are the same as they’ve ever been from the Waterfall method on down – Requirement gathering, Design, Implementation, Verification (Testing) and Maintenance.  Agile (in any of its forms) isn’t a silver bullet and doesn’t magic away any of these elements, it’s just a different way of mixing these basic ingredients to build the same cake.

To stretch the baking metaphor – proponents of agile will tell you that with a Waterfall methodology you might end up with a Dundee cake, when the users really needed an Eccles Cake.  This is exactly right, but the fundamental point is you’re only going to know the users wanted an Eccles Cake if you show them the first slice of Dundee cake, and they say “I’m sorry but you have misunderstood my requirement”.

So what? The point is that only the end-user can know if the product works for them.  However, often the end-user never sees the product until the end of a 12 month development phase, much like the Waterfall method.  What gives?

This problem is common in web-development.  You win a nice fat contract to build out a site for your customer (lets call them AcmeCorp).  The dev-team apply their shiny agile methodology and ship working code to AcmeCorp every few weeks.  AcmeCorp looks at it and supplies comments etc, you ask for clarifications, get more requirements and so on and so forth.

12 months later AcmeCorp launches their shiny new site, but it doesn’t get the traffic they expected, or users don’t like the features, or it doesn’t appeal to the target demographic.  Spot the problem?

You might as well have managed the project Waterfall style, you did 12 sprints/iterations/leans/kanbans/cynefines, but from the End-User point of view, it was a big-bang delivery.  You’ve been doing CAT (Customer Acceptance Testing) when you should have been doing UAT.  Arguably there may have been some benefits about getting feedback from your Customer, but if you are not getting feedback from End-Users as early as possible in the project, you’re asking for major trouble.

Conclusion

Agile teams must strive to get feedback from the end-users, not just the customer, as early in the project as possible.  As the feature you and the customer are convinced is the killer-app, may well be something your end-users don’t care about, or don’t understand how to use effectively.

UAT is the name of the game, and Agile teams should impress upon their customers the importance of putting the product under the noses of the real end-users as early as is feasible.