Skip to content

Improving Quality Up Front One Check-in / Build At A Time

December 27, 2011

One issue that many teams wrestle with is how to ensure that they have high quality code much earlier in their development lifecycle.  There have been many books written on the subject and I’m sure there are a few PhDs out there who could present a full twelve step program for helping to enable this to happen.  I thought that in this blog post I’d share with you two very simple things that one agile team did that greatly improved the quality of their code by simply expanding on and leveraging some of the tools and best practices that they already had in place.  The first was to automatically ensure code reads upon code check-in and the second was to implement a unit test harness that was part of both their source code control and continuous build systems.

First for some background:  Imagine that you are part of an agile team where 5 of the Fortune 10 cannot process an invoice globally without your system in place.  Your software solution has to hijack an invoice  in real-time, analyze the entire invoice along with every line item, and calculate the appropriate sales, use, and value added taxes in over 170 countries for thousands upon thousands of tax jurisdictions.  Then you have to modify the invoice and get it back in to the appropriate ERP system (SAP, Oracle, homegrown, etc.) in milliseconds and seamlessly such that quotes and invoices can be generated from those systems in a transparent way.  Your system is mission critical:  it is simply not allowed to break down or have errors that would cripple your customers’ ability to generate invoices or send out quotes.  Lastly, this same software is now enabled as a multi-tenant SaaS solution for the SMB market with even some of the biggest names in e-tailers on board.

One agile team that inherited this challenge knew that they had to be predictable in their product delivery commitments and that they had to have a very high level of quality.  Failure was simply not an option for this team if the company was to survive let alone thrive.  They knew that they spent a lot of time integrating components and modules (one installation was deployed on a cluster of over 32 servers) on the back end of the development process and that not catching quality issues up front made it almost impossible to track down issues later in the cycle or after the software was deployed or distributed.

Code Reads Prior to Check-ins:  One process that the team implemented was that every piece of code that was checked in had to go through a code read.  No exceptions…  They created an automation harness that made this very easy to both implement and enforce.  Upon invoking the command to check code in either from their IDE or the command line the system would automatically prompt them for who their “code read” partner was for this sprint.  A script would then automatically create a diff file between this version and the prior version then e-mail their code read partner a zip file containing:  the check-in comments, the original version of the source, the new version to be checked in, and the diff file that contained the differences between the two versions.  It was then up to the code read partner to either approve the change as being okay to check-in or to go back to the requestor with any comments, suggestions, and/or feedback in terms of making their code better.  Once approved the code read partner could then complete the check-in process and the version control system that was maintaining the source repositories would take over and do its normal work.  With each agile sprint the code read partnerships where changed up to keep things fresh.

Pros:  This really enabled the team to both implement and enforce their agile team policy that every change was to have a code read by a peer.  It allowed for the enforcement of coding standards as well as for a safety net in terms of ensuring that the code that was checked in was really ready for prime time.  It also ensured that knowledge of the internals of the system was broadly shared within the agile team vs. having a single guru or a handful of ninjas knowing everything about the system while reducing the need for tribal knowledge and folklore amongst the rest of the team.  Quality of the system also significantly improved as a fresh set of eyes often caught programming mistakes or were able to think of use cases, boundary conditions, or potential side-effects that the original authors may have overlooked.  Developers also became much more careful when making changes as they didn’t want to waste the time of their code read partners, which also improved overall code quality.  Lastly, the implementation was both automated and brain dead simple in both its scripting and the leveraging of the source code repository management system they already had in place.

Cons:  There were a few cons that the team had to work out as this was phased in.  First and foremost those developers who loved to check out tons of source files and go dark for a long period of time had to learn how to play nice with others.  Although this was a positive, it was initially a negative as the team worked through the organizational change management (no pun intended) issues.  They also had a number of situations where the more junior members of the team felt threatened by the more senior members as their code was inspected and this took some time to get over.  Where you had the case of code-read zealots, those who felt that only their own code was fit to eat off of, the team had to learn how and where to temper standards or approaches and those extremists had to learn that their way of doing things wasn’t the only way or the only right way.  Lastly, the scrum master had to ensure that there was time for the code reads in each sprint such that check-ins were not held hostage while waiting for their code reads.  The good news is that the time spent up front on checking in only high quality code more than made up for itself within each sprint as well as downstream.

Automated Unit Test Harnesses That Would Break Builds:  The other thing that this team did was implemented automated unit and system test harnesses, but in such a way that all in development became experts at this vs. a narrow group of quality assurance automation warriors.  Like their previous code read effort this was implemented in a very similar way:  Your code wouldn’t check in unless the updated test harness was created or checked in.  They tied this in to the version control system and continuous automated build system.   When you checked in your code you also had to check in the test harness that would exercise the APIs, objects, methods, web services, etc. of that module.  Without the test code being checked in you couldn’t check in your regular code as they were joined at the hip.  With the continuous build system these automated harnesses were automatically executed upon completion of the build.  If any of the automated harnesses failed, then the entire build failed, and as a developer the last thing you wanted was the team wide e-mail to go out that you had broken the build.

Pros:  It was amazing to watch how quickly the team went from having no real automated test suites to having almost the entire system covered in a rather quick period of time (about a year).  It was very easy for the team to build the test harnesses, check them in with their code, and have them plug in to their continuous build framework.  The team was rewarded with timely “do no harm” regression and performance tests that were automated.  It built a quality up front mentality within the team as they often coded the test harness changes first then worked on creating or updating their code.  This was used for all types of code, coding languages, and technologies from Java, C#, and javascript to database definition files and ETL source.  It also broke the mold that only QA automation engineers could build test harnesses as it empowered the entire team to solve the automation harness / framework problem as part of their normal engineering jobs (which allowed the QA Automation Engineers to focus on other high return efforts).  This effort quickly spread to other agile teams as a best practice and really helped prevent us from having what I call “self-inflicted gunshot wounds.”

Cons:  There were a couple of developers who decided to cheat and to check-in dummy harnesses that simply returned PASSED no matter what happened.  Let’s just say that the code read partners caught these rather quickly and the perpetrators were shamed in to submission by their teams. At first the team members assumed that this was going to place a huge and unnecessary burden on their team; the reality is that it didn’t add much time to developing the code behind a story and the time saved on the backend more than made up for the time invested up front.  There was also some resistance at first when it came to developers building automated unit test harnesses, especially from the QA team. However, in the end it worked out to where all parties were embracing the approach and championing its successes.

If you and your teams are exploring a “Quality Up Front” initiative I highly recommend these two best practices as part of that program.  They are easy to set up, administer, and execute on and they both have a very high ROI in terms of the time and money saved downstream as issues are caught and fixed much earlier in the development lifecycle.

This is a companion post to my Elevator Speech Series… For more on “delivering high quality solutions on time that are innovative, delight customers, win reviews, and disrupt competitors; all while having fun, being ethical and transforming the business” please see this series.


From → Quality, Software

  1. Great article and something every team should be doing.

    However I think you may need to create a new blog article called “moving quality even further forward”.

    It is sad, but I find many IT departments that still don’t use source control and even more that don’t have a continuous build environment.

    Hopefully we (IT people) have made some progress over the years. I haven’t had to tell people they need a development environment lately. 🙂 so things are moving in the right direction.

  2. I’ll admit it – from my product management perspective, I was skeptical when our team began code reads and unit testing. I mean that’s just going to slow things down and gum-up the agile process – right? Well – No.

    The first release was rough – some late hours and slim to fit trade-offs, but when it went out the door, it was right. The first significant release in some time that didn’t require a follow-up fix. Within six months we had the process down and our release cadence was more predictable. We ended-up delivering more value with fewer releases and no costly ‘do-overs’.

Trackbacks & Pingbacks

  1. Demystifying Agile – Rants of a Madman – Requirements and Acceptance Testing Should Not Be Akin To Pornography « markslawler

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: