Code Coverage, and false quality assurance

Today I got into a flame war after sending out an email asking the engineering team to look at integrating code coverage into their daily builds. Instead of rehashing the entire thing, I’m just going to cut and paste the emails that went back and forth…


Since we are discussing code performance metrics a lot, I would like to also bring up code maintenance/quality metrics as something that we have collectively failed to address. I would like to get an idea of our unit test coverage — code coverage is an oft abused metric, but it does provide a good starting point to measure how good of a job we are doing writing self testing, self maintaining code. I would like to address code coverage in the following areas: ..

Uber Developer X:

I think that is a great exercise for our two SDETs … I don’t really want to waste any
of my time with it.

It’s easy to do, however of questionable benefit. The people that are
able to interpret the numbers right already know their coverage and
those that aren’t will simply misinterpret them. Net gain: 0 .

WTF? I didnt know where to begin…after some time to calm down, I decided to write a calm, rational response:

{Name of Uber Engineer X},

Wow. So after my jaw bounced off the ground, and I involuntarily lost control of my leg muscles, and after my vision cleared up, and _only_ after I edited all of the initial four letter words out of my response, I can only say that I am in complete disagreement with your conclusion, and quite frankly I’m just stunned — and disturbed — that you, who are as metrics and measurement driven as anyone I have met, who I have a great deal of respect for as an individual and as an engineer, can take such a dismissive attitude towards code coverage.

Test coverage, as I’ve noted below, is an oft abused metric. In fact, there is enough evidence in the industry to show that 100% code coverage doesn’t actually prove anything about the quality of the code. So I’m not beating that drum.

However, code coverage is definitely relevant when the generated metric is being tracked — relative to itself — over time. In other words, seeing a drop in code coverage over time is worth investigating at the very least. Knowing where you have no coverage is useful information. NOT KNOWING any of the above is voluntary ignorance, and reduces the utility of any unit tests written.

So the sooner we start, the sooner we know if we have a gaping hole in test coverage, and where that hole might be, and the sooner we can determine where holes might be emerging (and address them). Again, for someone as fixated on performance and metrics as you are, it floors me — I am literally getting off the floor as I type — that you don’t see the benefit in starting to measure test coverage over time, immediately.

Because running test coverage tools is easily automatable, it seems to be very useful to take the time to do this between iterations, because it only needs to be done once to measure repeatedly. Net gain = very high.

What I really felt like saying was: hey, am I in a bad 80’s movie? Is it the distant and murky past, where testers are a lower form of life, suitable only for automating the works of the almighty software engineer? Or maybe I’m delusional, and I have been fantasizing that it’s actually 2008, and testing isn’t just for SDETs anymore, in fact the best unit tests are being written before the actual code gets written. That’s right, in this mythical shangri-la, software engineers actually take pride in writing and running unit tests, not leaving the testing to someone else. Because at the end of the day, it’s better to know you suck than to pretend you dont.

One Response to Code Coverage, and false quality assurance

  1. […] that is of limited interest. Who really cares what I think about technology X , or that I got in a flame war about Code Coverage? I think more people would be interested in what I’m using for code coverage, how I’m […]

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: