Not the blog, the code.

Among the many weird and wonderful iterators I have in my personal JGreenspun toolkit is one called the FlatteningIterator. It’s a useful little widget for recursively digging through mixed collections of objects and collections. Completely impossible to give a decent type signature to unfortunately, but what can you do? (I mean, I can conceive of a type system in which you can, but it’s basically going to have to be dependent typing). It forms a major chunk of the functionality for my recent WebCrawler implementation.

Indeed, that implementation is exactly what I want to talk about – the very first time I came to run the webcrawler it failed horribly. Rather than, as one might expect, being a failure in fetching or parsing the web page, the guilty party was my nice little utility iterator.

In my efforts to preserve laziness (that is, an object would never be fetched until it was needed), I’d basically screwed up. But I’d done it in a mildly subtle way so I never even thought to unit test it. The iterator’s hasNext() method returned true iff there was a non-empty iterator available to it.

Can you spot the problem? If the iterator available is an iterator which only returns empty iterators then the flattened iterator will falsely report that it has a next element, so using the class while(iter.hasNext() { iter.next(); } loop will throw an unfriendly exception. Unfortunately I completely failed to spot this until it came up in actual practice.

Thus, even though I had extensive unit tests for this class, they missed the one way in which it was actually broken. (The class passed all the other unit tests first time). This is because logic errors in the implementation often correspond to failures to spot an edge case. So if the same person doing the implementation is writing the unit tests, the unit tests probably won’t cover that edge case either.

Fortunately it only took a little time to figure this out, and once I had the unit test was updated to cover this case. So, no real harm done. But I’d have been really annoyed if it had taken me a long time to track down a bug and it turned out to be in code I thought was safe due to unit testing.