Note: I originally posted this on WordPress on April 2, 2012 and have updated with some modifications in this refresh…
How many times have you downloaded an update to your latest app or gone to a vendor’s web site and nearly entered a fugue rage state because something that use to work is now broken? How many times have you cursed slower performance that isn’t a simple glitch, but seems tied to the latest release? I know I’ve experienced both; it is very frustrating and every occurrence undermines my confidence in those products and erodes any good will I might feel toward the solution or the vendor. I have to admit that as a software professional I have unfortunately been the cause of this in my own career as a developer. All of my teams have also done so at one point or another.
This is why I adopted a simple golden rule many years ago: Do No Harm.
To me one of the worst sins in software development is to introduce what I like to call a “self-inflicted gunshot wound” in to your product. It hits home once the release is deployed in to production and you start hearing about escalations regarding functionality that use to work. These issues can be both in terms of “you broke it” and “performance now sucks”.
It ruins everything: All that hard work you put in to adding some great new feature is now lost as the customer can only focus on what you broke.
I believe that before any code is pushed in to production the team should be able to answer these questions:
• Will you cause harm in this release?
• Have you broken something that use to work?
• Is performance at least as good as it was in the previous release?
At some point in my career I started asking these standard “Do No Harm” questions at every “Release to Manufacturing” checkpoint meeting. The purpose of these questions were to be very clear in terms of what was important to me–in addition to delighting our customers with “Gee I’d Buy It Just For That!” new features and capabilities I also expect a customer’s confidence in the release not to be undermined by discovering regressions.
When I first started asking these questions I’d get blank stares in return. Over time I started to be assured that everything was okay, but when I asked “How do you know?” I’d get folklore as to why folks believed their answers. Nobody really liked my follow up statement to this “wish it and it will come true” positioning: “Prove it…” Eyes would roll and folks would complain that I was changing the rules to the test; this expectation had never been set by previous leadership and it wasn’t fair for me to ask only at the final checkpoint meeting before a product was released.
They were right–it wasn’t fair that it was being asked only at the very end… At the same time the issue wasn’t with me nor my questions… The real issue was that the teams were not asking themselves these questions throughout the entire software development cycle. I didn’t let this sway me and I kept asking and asking and asking.
Progress! Within just a few months all of my teams began showing me their test coverage over existing functionality (some even with dependency charts that showed where changes were made, the risk those changes posed to other areas of the code, and how those areas had been focused on for regression testing/scripts). They would also come to me with their performance test results such that they had sufficient proof. Did that get rid of these issues in production? Not entirely, but it did greatly reduce them.
The miracle came within about twelve months as the teams began to appreciate how bad regressions really were… With every build many teams now knew the answers well before these final hardening sprints and checkpoint meetings. Some had even decided that as part of their continuous build processes that the appropriate unit test harnesses would execute both regression and performance checks; failing the build where appropriate and becoming a core component of their “Quality Up Front” initiatives.
In the end the teams’ solutions became more reliable and customers were not being negatively surprised. The teams were spending more time adding new features vs. working on emergency patches or spending time catching up with what they had broken in the previous release. All was well in Whoville.
Does your team suffer from self-inflicted gunshot wounds in production? If so, does your team have a “do no harm culture”? If not, is it worth it to start asking these three questions with each and every release? With each sprint? With each code checkin? Try it…it worked for my previous teams; I hope it will work for yours.
Update: I’ve adding a link to a whitepaper called “Girls in IT” at the bottom that not only highlights the issue, but what can be done to help solve it.
After reading a lot of nonsense on this topic, I came across this great article by Hadi Partovi, who is a co-founder of Code.org. As opposed to diving in to stereotypes, like other posts and articles have done over the last couple of weeks, Hadi cuts to the real reasons. The first reason he cites as being the primary cause: Computer Science is not taught in US schools! Duh… He’s so spot on I don’t know where to begin! I’ve worked with lots of young women as part of TechStart Education Foundation’s Oregon Game Project Challenge and girls are interested in learning to program in K-12. One key problem is that by 7th and 8th grade there are just not that many opportunities to get plugged in.
Please give this a read and then learn more about how to support #STEM programs and code.org in your local K-12 classrooms.
Here is a whitepaper called “Girls in IT”. It talks about the shortage of women in technology and trying to narrow the gap. http://www.bespokesoftware.com/wp-content/uploads/2013/12/GirlsinIT_white_paper.pdf (via @ShimCode).
Rants of a Madman – Agile and Performance Objectives – Only Pigs Allowed on the Field
I’ve had the pleasure of working in IT twice in my career these last 25 years. Both times were short stints as I came to my senses rather quickly. I love IT and what they do to help enable the business, but I hated being there. Each time I left the building running as fast as I could, screaming from the top of my lungs with joy, vowing I’d never do it again. There is just something about being in the development ranks within IT that is night and day when compared to working in similar positions for a commercial software company. I would say that if you work as a software development professional within IT you should run, not walk, to a commercial software company to see how the other half lives. You’ll never feel better. Here are just a few reasons off the top of my head:
• You are a cost to the business vs. a revenue generator
Let’s face it, in IT you are a cost. A cost the business hates and often considers a tax. It cannot fathom the value it receives for the allocation $ that are an uncontrollable line item in their cost centers. They know in their heart of hearts that if they were given those $ directly and hired their own technologists they’d get far more done, quicker, better, cheaper.
The reality is that you are not the primary force behind the product that is being sold to generate revenue for your company. The real stars of the show are the product folks—they are at the center of the business universe. You are simply a cost that helps the back office work more efficiently. You are seen as the annoying gum on the bottom of the business’ shoes. You are a cost that needs to be controlled and reduced.
In commercial software the roles are reversed. The technology is the product and is the revenue generator for the business. What you create is what is valued by all organizations across the entire business. Yes, you may even be considered kings in some of these elite software organizations. There is an IT group where you work, but they are the allocation on even your own cost centers as they work in the background to help the back office flow smoothly.
• You don’t really have a customer
In IT your customer is really difficult if not impossible to identify. Often it is any business user with an opinion, right or wrong, timely or Johnny-come-lately. As a technology professional you are also set up for failure as you don’t have a Product Manager or true Product Owner. In IT you might be lucky enough to have a business partner who is supposed to be the voice of the customer, but in reality you find that they don’t speak as a single unified voice. You find that they simply represent an opinion and that opinion really doesn’t carry much weight beyond perhaps their own group or department. When trying to map their perfect world on to other departments when it is time to do a rollout you find that they lived in a fantasy land. Their fault? Of course not; you built the solution and you spent/wasted their money doing so!
In commercial software it is really easy to identify your customer. They are the ones who reach in to their wallets and pay for your products as well as any subscription or maintenance fees year after year. By the time they reach for their wallets they are eager to consume what it is you are providing. If they continue to pay for your solution it is because they are truly delighted by the products you have provided them. When creating your product you have the strength of a Product Management team and Product Owners who truly represent the collective voice of the market and the customer. They know that they cannot please everybody, and that everybody’s individual opinion or request isn’t a show-stopper. They know that they have to create the best solution for the masses and not just for Bob or Sally down the hall. They know that part of their full time job is the juggling of conflicting input and at the end of the day making the call as to what is the right thing to build for your business to prosper.
• You live in the land of customization
In IT you are always building a customized solution or modifying something in to a customized solution. Let’s face it, your business users are the amazing experts at their own home-grown back office business processes. It is your job to make sure that your software is customized to match that industry best practice process that Bob and Sally invented while hunkered in their bunkers each day. As an aside I must admit I laugh each time I see how many tens to hundreds of millions of dollars were wasted on a failed ERP implementation. Often the culprit was all of the customizations that either broke the solution or didn’t scale beyond the pilot deployment group. How many times would the implementation have succeeded at a dramatically lower cost had the solution been rolled out vanilla, with Bob and Sally either been trained how to do their jobs differently (or replaced with Tom and Mary who were not locked in to their home-grown process)? This is the life of an IT software development team—building and customizing to whatever spec Bob and Sally demand regardless if it is the right answer or not.
In the commercial software space you are creating a solution that is intended for a broad audience out of the gate. Where there are customizations, you have enabled those either through self-service configuration screens and options or through open APIs and Web Services. Your solution isn’t intended to simply automate what Bob and Sally invented, it is to bring to them a best in class answer that leverages the best practices that you have seen across an entire industry. It is to take what is more efficient and free their time up to do other things for their business. It is to delight them with these better ways of doing things vs. doing it their particular way.
• You work on single event projects vs. products
As a software developer or QA professional within IT you are often working on single “fire-and-forget” projects. The business sees these as one-off efforts that will help solve a particular problem the business has in its back office workflow. When funded, these very rarely contain any funding for future sustainment work. The assumption that is often made is that if the solution needs any updates, they’ll just go out and get funding down the road for another project to do that work. That or IT will simply “eat” the costs. Another strategy employed is to try to “get the world” while you work on this current project. Where they do fund sustainment it is often with only a skeleton crew who is doomed to triaging every conflicting input from rather opinionated end users. Who knew Bob and Sally had relatives?
As an IT resource working on fire and forget projects you are also considered to be a fungible resource. Have an extra 3 hours on Thursday? Go work with Team W and knock out one of their stories for them even though you never heard of the project before vs. refactoring something in your current project. Context? Who needs time for gaining knowledge or context switching? You just pump code or QA it, right? You are simply a nomad who wonders from project to project, being summoned for your domain knowledge on a past project you worked on over five years ago and are expected to have instant recall. You are expected to help without impacting your current project.
In commercial software you are normally working on products vs. one-off projects. These are solutions that have a lifecycle and multiple versions. In their releases and even in their Agile sprints the teams and management are aware that they need time for new features, time to refactor old code, and/or time to do sustainment work on features already in customers’ hands. They know that they cannot shove everything in to the current release and that they have future release to do something even better, based on real customer feedback vs. trying to guess everything right the first time. They know that they can use multiple releases to phase in major architectural changes or a major feature that is “turned on” a release or two downstream when it is ready for consumption. These product teams are a highly collaborative unit who continuously ratchet up their best practices.
• Time tracking
In IT the business doesn’t trust you. They think you are a group that is overfunded and constantly under deliver. Because they wonder what you do all day they make you track your time. If they don’t make you track it, then the CIO does so because he/she has to justify how you spend your time to their untrusting business partners. I even worked at one place once where you had to account for every 10 minutes of your time on a timecard and they paid auditors to walk the halls to ensure that each and every timecard was up to date vs. something you filled out at the last minute on Friday.
In commercial software, if you track your time it isn’t because the business doesn’t trust you. It is to help become better estimators over time. It is to help understand what your true capacity is such that they can help you. Quite a different dynamic at play.
• Fixed bid projects with open ended requirements
In IT you participate in a very crazy yearly budgeting process. The business wants something automated and attempts to create a project to make them more efficient at something. As such they ask IT leaders to “name that tune” very early in the process often based on nothing more than a concept. Of course the $ and hours quoted are too high and they don’t believe you as you cannot prove it’ll really cost that much (due to their own vagueness in what it is they want). They squeeze you until the stone drips blood and then they run off to secure funding. If they receive the funding it was less than even your bargain basement estimate. Once the project is underway you find out that the go-cart they received funding for needs to sleep 6, have a bathroom, running water, electrical, WiFi, and can tow a boat. You argue that the requirements are changing and will cost more in time and money. They argue back that nothing has changed: you always knew they were funding a vehicle. Go figure…
The other thing that can happen in IT is, because of how projects are funded, they know they only have you held captive for a fixed amount of time. In a world where sustainment costs are often not modeled the belief is that if you don’t make your 1.0 release the equivalent of a 3.0 that they’ll never see you again and they will never get those future downstream features and enhancements funded. It is also the opportunity while you are being held hostage to sneak in any tag along hitchhiker features they can think of. Of course none of this was taken in to account or budgeted for, but then again they are paying you too much for this project so you just need to suck it up.
In commercial software the organization knows that you only have four levers: scope, quality, budget, time. You also know that quality should really have a fixed position (something rarely taken for granted in IT) and that budget and time are often linked. That leaves scope as the main or only lever that can be moved. Because you have a Product Management team that understands the market, has modeled the competitive landscape, knows what customers want to pay for, and has modeled the revenue stream they know how to prioritize what needs to come first. They know what needs to be gold plated and where something else needs to simply be a checkbox. These ninjas ensure the teams are working on the right things for the right reasons. They may not like the trade-offs due to your technical team’s capacity during a release, but unlike IT business partners they don’t always try to play the victim. They may push you, but they are not constantly trying to cram 25 pounds of potatoes in to the 5 pound sack that was funded. Why? In commercial software they go down with the ship right along with you whereas in IT they can fire shots in to the boat without care as they are standing on the dock.
• Elusive ROI calculations
In IT trying to get a project funded is next to impossible. Quite honestly the business has a very hard time coming up with any form of ROI calculations that justify the funding of any technology effort. One of the primary reasons for this is the business has no problem looking at ROI when it comes to IT cost savings (after all you are a tax that needs to be reduced), but it really wrestles with ROI that involves signing up to their own operational savings and efficiencies. Good luck trying to get many business sponsors to model how much more productive the business can be with this solution in place, how many fewer people would be required, or how it can save money in the future by reducing their own departmental budget needs. Go figure…
In commercial software the ROI is pretty simple to calculate given the technology is the product that is sold to customers. Depending on the maturity of the product, it needs to generate between 7 and 10x in revenue over the cost of developing and maintaining that technology each and every year. Why that high of a multiple? It is simply the inverse of its expense plus a margin when compared to the rest of the company: R&D is an expense and what it creates has to also cover sales, marketing, finance, HR, operations, and some margin for the company. In professional services organizations the return is a much lower target (sometimes even as low as 1.3x). It is much easier to run the numbers backwards even when looking at individual product feature enhancement requests in commercial software than in IT. You want to delight your customers, but at the end of the day you can only do so if you are still in business. This fact of basic survival keeps the math pretty simple.
• Lousy pay
Whether software development talent in IT wants to admit it or not, there is a caste system. At the end of the day you most likely make less than your counterparts at commercial software companies. This is for several reasons. The first of which is commercial software companies strive to hire tier 1 talent; they want their people to be so good at what they do that they indeed command the 75th percentile or hirer in compensation. They are looking for folks who have computer science degrees and years of experience being on successful commercial software teams. Those folks who work at commercial software companies on these teams really don’t apply for similar software jobs within IT. They love working on teams who create products that delight their customers day in and day out and tend to go to other commercial software companies.
Because IT is a cost center relegated to the operations of the business it is often looked at as a group where costs must be tightly controlled. If this means taking knowledge experts and trying to teach them how to code, then so be it. If it means hiring people in the lower quartiles of compensation, then so be it. It is also a group where commercial software development talent really doesn’t apply for these positions; why would they?
• Stale talent with too much specialization
In commercial software folks tend to move around much more, perhaps every 3 to 5 years. Those who work at startups tend to move around even more so as business models more often that not fail regardless of how good the technology is (the case of building the wrong thing well). That said, because commercial software development talent moves around more than their IT counterparts they are exposed to numerous best practices. They also tend to stay plugged in to technical communities, user groups, and forums as they are great networks to leverage when it is time to land their next job.
In bloated IT organizations you also have the issue where job specialization actually harms individuals and their resumes vs. helps them. In IT, for example, you may have very distinct job descriptions for Data Modelers vs. those who create Data Definitions for relational databases, manage databases, create SQL, optimize SQL, create Extraction Transformation and Load (ETL) jobs, and who code and test transactions to those databases. That over specialization in IT limits their ability to apply for and succeed in commercial software jobs.
In commercial software those same eight separate job descriptions in IT could easily be covered by one or two Sr. SQL Database Engineers. In commercial software there is specialization, but knowledge workers end up having broader skills even within a specialization.
• The business hates you for other reasons too
Here’s a dirty little secret: In IT as a software development professional you often make more than many in the business do, which fries many of them. This simple dynamic of knowledge worker supply and demand and the value placed on technical talent often drives your counterparts in the business nuts as you estimate projects for them. I’ve even seen it drive professionals in service organizations such as finance and HR bonkers and they should know better. Many believe that by being in IT you are simply doing back office work; how is it possible that your folks may make more than many of theirs do? They believe that product people should make more; in their eyes it just isn’t fair.
In a commercial software company this underlying resentment against the cost of technical talent just doesn’t exist. Because the software development teams are responsible for creating the products that the company sells you just don’t find it amongst business partners and other functional groups. If it is there, it isn’t out in the open as a festering wound like it is in IT.
Quite the list, no? Don’t believe what I’m saying? Simply ask somebody who has worked in both types of industries. I’ve only sampled some of the differences off the top of my head—there are many, many more. I’ll let those who feel like commenting add what they’ve observed.
So, are you currently working in IT as a software development professional? There is nothing wrong with that and your business really needs your technical talent whether they think so or not. You should take pride in the value that you add to the business each and every day. That said, if you are feeling underappreciated, there is an entire commercial software industry that gets what it is you do, why you love to do it, and it truly does value you. It may be difficult to break out of IT and join the commercial software development ranks, but if you can you’ll be glad you did. I’m glad I’m back!
One of the best articles I’ve ever read on this topic. Cory House pens a great piece on why, as developers, writing clean code really matters. In one section he compares developers to authors and makes the assertion that each line of code one writes is likely to be read 10 or more times by others during its lifetime. That’s probably 9x more than this blog! ;O) Enjoy the article here… It is a gem…
What did we do before Google came around to remind us of all the great events in history? I am glad they called out today was the 150th anniversary of Abraham Lincoln’s famous Gettysburg Address, but am disappointed at the same time they chose not to have a Google Doodle for such a historic event. That said, here is what Google shared: “150 years ago a 2-minute speech shaped a nation“. A lesson in how such few words, when penned and delivered by a master orator, can have such a huge impact…