Some stuff on example simplification in Hypothesis

One of the slightly thankless tasks I engage in in Hypothesis (subtype: People only notice if I don’t do my job well) is example simplification. I spend a lot of time on example simplification – far more than I think anyone else who writes a Quickcheck bothers with.

What is example simplification? Well, when Hypothesis tells you that an example that breaks your test is [1, 2, 3] then it almost certainly did not start out with that. It probably started out with a list about 20 elements long with numbers like 349781419708474 in it, and then prune it down to [1, 2, 3].

Example simplification is important because random data is very messy but there is usually a simple example that demonstrates the bug, and the latter is helpful but the former rarely is.

Simplification is of course not new to Hypothesis – it’s been in Quickcheck for 15 years – but Hypothesis’s example simplification is atypically sophisticated.

So, why so sophisticated? Why is the normal Quickcheck approach not good enough?

Well the official reasons are as follows:

  1. Hypothesis generates much more complex examples than Quickcheck. e.g. Quickcheck will generate a long list of small integers or a short list of large integers, but will never generate a long list of long integers.
  2. Tests in Hypothesis tend to be slower, due to a mix of Python performance and Python idioms.

The unofficial reasons are probably actually closer to being the true reasons:

  1. Including bad examples in documentation is embarrassing
  2. My test suites do a lot of example simplification so faster example simplification gives faster test suites gives a less stressed David
  3. I find combinatorial optimisation fun.

At any rate, the result is that I’ve spent a lot of time on this, so it seemed worth sharing some of the knowledge. I’ve done this in two places. The first is that I gave a talk at the London Big O user group. The slides are also available separately.

I was going to transcribe it, but actually I didn’t think it was a very good talk (I prepared it in a bit of a rush), so instead I’ve built an ipython notebook working through the concepts. This actually goes slightly further than the talk, as I figured out some things at the end that I need to merge back into Hypothesis proper.

This entry was posted in Hypothesis, programming, Python on by .