Skip to content

A Lesson In Resiliency

February 27, 2016

Back in the late 1980s and early 1990s there was a huge industry rivalry between two powerhouses in the personal computing software utilities sector. Up in Portland, Oregon was Central Point Software and down in Santa Monica, California was Peter Norton Computing. Both companies were bitter rivals as they fought a heated head to head battle to be the top dog in this market, slugging it out for revenue and market share. Central Point Software was on a path aimed towards filing an Initial Public Offering while Peter Norton Computing was stolen off of the launch pad, acquired by Symantec. That event didn’t change that the newly renamed Peter Norton Group was out for blood; their goals were clearly stated as “Delight customers, win all reviews, bankrupt competitors” — even the janitors knew it.

Both companies produced product suites which competed head to head in a daily battle to win both new customers and to take each other’s customers away. In one corner was PC Tools, the Central Point product, and in the other was Norton Utilities, from the Peter Norton Group. One of the main battlegrounds was in the area of disk optimization. Back when dinosaurs roamed the Earth—okay, back when personal computers had 12 Mhz Turbo switches and 10 to 20 MB hard drives—computers would get extremely slow if the hard drive became fragmented. Compress, from PC Tools would optimize the drives by rearranging file sectors to be contiguous again as would Speed Disk, the Norton Utilities product. This would make the computer fast again as the head on the hard drive wouldn’t have to move all over the place to find file sectors to read a file into the computer’s memory. One could argue which product was better than the other. Product reviews would tend to focus on which product would complete before the other as well as which drives would perform faster after being optimized.

Let me now take a brief pause to introduce you to Brad Kingsbury, the first programmer hired by Peter Norton and one of my earliest mentors…

Brad wrote the very first versions of Disk Doctor and Speed Disk at Peter’s kitchen table. He knew it was time to go home from work when Peter’s wife was ready to set the table for dinner. Brad was obsessed with clean code that performed both fast and flawlessly. Brad worked with my team on Norton Desktop for Windows and was my boss on Norton Antivirus in its very early days before it hit the big time. He was a perfectionist and expected all of his developers to know exactly how many clock cycles every line of code written in C would compile to. He also expected each of us to test every code path while in our debuggers, shoving values in to memory (or even registers or on the return stack) if we had to in order to ensure we could single step our code through every possible conditional. To have a defect found in your code was a badge of shame. Anybody who wasn’t out to make their product both fast and perfect he crowned a Slacker, a badge of shame you worked your butt off to shake. When Microsoft told Brad that it would be impossible to write a version Speed Disk for Windows due to how the operating system worked and handled disk i/o operations, Brad scoffed and came back with a working proof of concept version after almost a week of straight all-nighters.

Now back to my story…

Both PC Tools and Norton Utilities were in a dead even heat in the marketplace. Both had similar products and very loyal customers who would align with their favorite. The market had become saturated and the only way to grow revenue was to take customers away from each other. This created a fierce feature battle between the two companies. Despite the pressure to add more and more new features, Brad would not back off of his mantra regarding being able to delight the customer, but also doing so with the fastest and most reliable products. Brad knew in his heart that our products were engineered better than the competition and that is when he discovered a fatal flaw in the PC Tools products that Marketing exploited in a way that nobody could have imagined.

There used to be an industry trade-show called Comdex, which was held each year in Las Vegas (today the Computer Electronics Show picks up where Comdex left off). In the Symantec press suite two identical IBM personal computers had been set up next to each other. Both had matching hard drives, which were exact duplicates of each other. They invited the press to watch as they kicked off the presentation and demo. On the first machine they started by installing PC Tools (from floppies no less) and on the second they installed Norton Utilities. After the products completed their installations they then launched the PC Tools disk optimization product on the first machine and at the same time on the second machine they started Norton Speed Disk. A couple of minutes later as the presenters were talking a loud gasp filled the suite.

After both products had run side-by-side while they talked the two of them did the unthinkable: they ripped the power cords out from the wall sockets. There was no stopping the programs, no powering down, just the hard pull on the power cords. The audience saw the monitors go dark as they heard the very audible sound of the disk drive heads falling onto the storage platters and bouncing. The only thing louder in the room were the gasps that followed from the audience.

Both PCs then had their power restored and they began their boot sequences. The first computer, where the PC Tools Compress disk optimizer had been running, wasn’t able to boot as the BIOS claimed that disk had a fatal flaw and was unmountable. The second computer, where Norton Speed Disk had been running, booted up fine. The command CHKDSK (which was used in DOS to check the integrity of a disk drive) was executed and said everything was fine. The demo then continued on the second machine: Speed Disk was restarted and it automatically began its task right where it had left off when the power cord had been unplugged. It completed a couple of minutes later with a fully optimized hard drive.

Now the first machine, which had been running Central Point’s PC Tools, was in real trouble. Not only were the contents of the drive damaged so badly the computer wouldn’t boot up, but the PC Tools emergency rescue disk wasn’t working. When the demonstrators rebooted the PC with the emergency disk in the floppy drive, it couldn’t even find the C: drive to try to repair it. Everybody thought the first PC was now a doorstop.

The next part of the demonstration was priceless. They then proceeded to reboot the first PC with the Norton Emergency Rescue Disk, which worked, of course. The Rescue Disk repaired the hard drive on the first machine, previously damaged by the loss of power during the PC Tools Compress optimization. The demo then continued on by optimizing the first machine with Norton Speed Disk.

This demo was repeatable over and over again and without any variation in the outcomes. It became a huge buzz at the national convention and the press had a field day. For about a year the media decided to cover all of the stories they could find where PC Tools customers suffered outages and data loses while using their products.

It was just a couple of short years later that my plane landed at PDX for the first time. PC Tools had pulled their IPO, we had taken a huge percentage of their customers away from them, and Symantec had decided to purchase what was left of the company to consolidate market share. My job on the ground was to meet with the former Portland-based PC Tools software development teams and to help them learn “the Norton way.” I then went on, as part of my responsibilities after I relocated, to take over their security product teams and fold in the development teams from the newly acquired Fifth Generation Software company.

So….

Why am I sharing this story? I feel like I’ve seen this movie before, but many in this industry are doomed to not learn from its lessons and film the bad sequels. As software developers it is one thing to complain about how crappy the infrastructure is that your system is running on (i.e. how it isn’t fair that somebody pulled the plug out of the wall), but it is another thing to design and develop software with failure in mind. Coding and testing for “happy path” is a recipe for disaster even in advanced PaaS environments like AWS and Azure. We need to keep in mind delighting customers includes resiliency, as that is a critical piece of the equation.

Will this tale change how you think about designing, coding, and testing for failure? Do you already do this? If so, please share your insights.

Advertisements
One Comment
  1. Padraic permalink

    It reminds me of when I used to code slot machines. Resiliency was priority number one for all the reasons I’m sure you could imagine.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: