Skip to content

Rants of a Madman – Did you do harm in your last release?

August 2, 2012

As software professionals we take great pride in our work.  We are very creative with many of us feeling that the display is our canvas, the keyboard our brush, and that our works help our users accomplish more and in easier ways.  That said, how many times have you been frustrated when you’ve downloaded your latest phone app or gone to a vendor’s web site and have nearly entered a fugue rage state because something that use to work doesn’t anymore?  How many times have you cursed slower performance that isn’t a simple glitch, but seems tied to the latest release?  I know I’ve experienced both.  I have to also admit that I’ve caused both in my career as have most of my teams at one point or another.

This is why I adopted a golden rule many years ago:  Do No Harm.

To me one of the worst sins in software development is to introduce what I like to call a “self-inflicted gunshot wound” in to your product and not even know it.  Once it hits production you start hearing about issues in functionality that use to work.  These issues can be both in terms of “you broke it” and “performance now sucks”.  It ruins everything: All that hard work you put in to adding some great new feature is now lost as the customer only focuses on what you broke.

I love Agile, but there is one aspect that bugs me (okay I have several that bug me and this is but one of them): the belief some Agilists have that at the end of every sprint all teams can push their code in to production.  I do believe when you die and go to Heaven that is a level of Nirvana that you should bask in, but the reality is that many teams on projects larger than two guys and a dog making simple changes just don’t experience it in real life.  Code is complex and often if you tug a thread on one end other threads can get pulled on elsewhere.  Some teams keep trying to live up to the perfect sprint end-state regardless, but after enough self-inflicted gunshot wounds and angry mobs of users demanding their heads be placed on top of spires for ruining their lives they most often give up.  They slow down and introduce hardening sprints or other ways to ensure they didn’t break what was there before.

As the Scrum Master, Product Owner, head of development, or <pick your title here> I believe that before any code is pushed in production a team should be able to answer these questions:

  • Will you cause harm in this release?
  • Have you broken something that use to work?
  • Is performance at least as good as it was in the previous release?

At some point in my career I started asking what became these standard Do No Harm questions at every “Release to Manufacturing” checkpoint meeting.  Now don’t go rolling your eyes and start shouting “Down with Waterfall! Down with Waterfall! Down with Waterfall!”  I don’t care if you follow waterfall, Agile, or your favorite rapid hacking methodology I believe you need a checkpoint as a sanity check before you deploy your wares on to the masses.  The purpose of these questions was to be very clear that in addition to delighting our customers with “Gee I’d Buy It Just For That!” new features and capabilities that I expected to not ruin that delightful feeling with code that didn’t work.

In the beginning I’d get folklore as to why folks “believed” their answers to these questions.  Nobody really liked my follow up statement to this “wish it and it will come true” positioning:  Prove it…  Eyes would roll and folks would complain that I was changing the rules to the test.  That this expectation had never been set and it wasn’t fair for me to ask only at the final checkpoint meeting.  They were right…

So at first these questions didn’t stop the releases.  The teams needed time to digest them and internalize how they’d answer them in future checkpoint meetings.  That said they did make an impact as I kept asking and asking, never giving up on them.

In addition to me asking the same questions like a broken record it only took a few of these self-inflicted gunshot wounds in production, and the pain of managing upset customers and derailing product efforts to do emergency patches, for the teams to realize that perhaps they should be able to prove the answers to these questions before deployment.

Progress!  Within just a few months all of my teams would walk in to their final checkpoint meetings before release or deployment and show me their test coverage over existing functionality (some even with dependency charts that showed where changes were made, the risk those changes posed to other areas of the code, and how those areas had been focused on for regression testing/scripts).  They would also come in with their performance test results such that they had sufficient proof.  Did that get rid of these issues in production?  Not entirely, but it did greatly reduce them.

The miracle for me came in just twelve to eighteen months later…  With every build many teams now knew the answers well before these final sprints and checkpoint meetings.  Some had decided that as part of their continuous build processes that every night the appropriate unit test harnesses would execute both regression and performance checks; either failing the build.  The upside was that the team had immediate warning as to when something that had changed that day had caused such a negative impact to their solution.  Because of the timeliness of the warnings, diagnosing the causes were quick and the fixes relatively easy to implement.  It became of core component of their “Quality Up Front” initiatives.

In the end the teams’ solutions became more reliable, customers were not being surprised, and schedule predictability starting soaring through the roof.  Teams no longer were worrying about regression failures and performance issues showing up too late in the cycle that would impact the schedule.  All was well in Whoville.

Does your team suffer from self-inflicted gunshot wounds in production?  If so, do they have a “do no harm culture”?  If not, is it worth it to start asking these three questions with each and every release?  Try it…it worked for my teams; I hope it will work for yours.

Advertisements

From → Agile, Quality, Software

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: