Have a test suite

Two tests failed. They were rather benign tests too. I think I had put those tests there more for completeness than for some edge case I’d run into. The tests were for cases when some of my algorithms would produce an empty data structure. Almost as an after thought I was also saving the empty data to file using h5py – a wrapper around the HDF5 C libraries. I was moving code to a production machine, and those two tests failed.

The fact that in both those cases the data structures were empty did not strike me at first. I spent most of my time making sure setup.py was correct and I had indicated the versions of the packages I had on my dev machine and that the production machine indeed had at least those versions of the packages.

One of the things about Python is that the neat package manager works best for pure Python packages. Once you have packages that wrap libraries written in other languages which have to be compiled separately, you have to rely on the Python package maintainers to detect and flag buggy versions of the underlying libraries. It’s easy to forget this and be lulled into a sense of security once you have your setup.py file all annotated with required dependencies and you are sitting smug and warm in your insulated virtual environment. You forget that the cold clammy claws of the C++ ecosystem’s madness are never too far away.

After a lot of madness of my own, looking over the failed tests and the debug output and some increasingly desperate web searches I came across the following hit:

Screen Shot 2015-10-26 at 5.51.24 AM

Finally, I stopped looking at the Python ecosystem and looked at the libraries. The HDF5 version was indeed at 1.8.4 and upgrading to the latest library version made the test errors go away.

And I sat there for a while thinking how dangerous it is not to have a test suite. If I didn’t have those tests in there this bug would have been sitting in the code, quietly waiting. It’s not even a bug in the code I wrote. It’s an anonymous bug, from some unknown faceless colleague. Actually, it is code from a consortium, who are pretty well funded, since the library is well regarded and well used. And I would never have known until something blew up.

I mean everyone says you should have a test suite, but no one ever gives points for it. People look at how fast your code is, perhaps at the algorithmic tricks you use, perhaps at the nice UI you present, but no one ever says, Hey that’s a nice test suite you have there.

But that’s probably the only important thing in the code.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s