Or: Why missing visibility is hazardous to your tests
Some weeks ago, I was talking to a colleague about the merits of test driven development and its challenges in an embedded environment, such as limited memory resources and hardware dependencies. He was not happy at all about the way his team handled unit testing. They had a fair number of tests, but those tests were only run once a day. The tests were not run on the development branches, but only on the integration mainline. It took more than 30 minutes to run all the tests, and there was no separation between fast and slow tests. There was no direct feedback to the developers about failed tests. In fact, even on the build server, it was hard to find out whether any of the tests had failed. As a result, failed regression tests went unnoticed for a long time, and finding the cause and fixing a failed test was a lengthy and tedious task.
In our discussion, we quickly realized that none of the problems were due to the embedded nature of the project. One cause of the poor test quality was a simple lack of education: the members of the project team did just not know how to write fast and efficient test cases and integrate them well into the build process. Another cause was lack of visibility.
I might out myself as hopeless old-timer here, but at the beginning of my development career, WYSIWYG (what you see is what you get) editors where all the rage. The display on the screen was very close to the result on paper, saving some surprises when printing a document. Nowadays, this is not much of a selling point. But regarding unit tests and their visibility, we might turn the acronym on its head: What you don’t see is what you don’t get.
When a problem is in plain sight, like a pile of garbage outside your front door, then it is pretty hard to ignore. Being highly visible, we will be constantly reminded of the problem and the chances are high that eventually someone will take responsibility and address the problem. The visible garbage is likely to be removed. If, on the other hand, the garbage is stacked away in a shack or buried in the ground, no one will take offense and slowly people will forget about it.
What you don’t see is what you don’t get. If you don’t see the garbage lying around, you won’t get the garbage removed. And if you don’t see the quality of your software, you won’t get quality. Only if failed tests are plain to see, people will react and fix those bugs. Giving failed tests more visibility also gives them more weight, gives them more importance. It might not stop people ignoring them, but it will make it considerably harder.
Here, of course, is where build servers and continuous integration come into play. Build servers like Jenkins can monitor the source repository, build and test each commit, publish the test results and actively notify the involved people when a build fails. Especially the notification part is very helpful, providing active feedback as soon as some problem is recognized. I used to think that TDD was good personal programming practice and could be applied without continuous integration. But this is only true during the initial development stage, where the tests are used to drive development. Later on, they serve as regression test, making sure no regression creeps back into your code. So nowadays I fully agree with Jeff Langr (“Modern C++ Programming with Test Driven Development”) that continuous integration is vital to your testing effort.
So tune your build server and make sure those test results are highly visible. Because missing visibility is hazardous to your tests. And start seeing what you get.