Falsifying hypotheses in python

I haven’t written any Scala in ages. Mostly I’m OK with this – there are still a few language features I miss from time to time, but nothing too serious.

One thing I do often miss though is Scalacheck. It’s really the nicest instance of Quickcheck style random testing I’ve seen (this is admittedly not based on a great deal of experience such libraries other than Quickcheck itself and some exceedingly half-assed ports).

One of its nice features is that as well as randomly generating test data, it also minimizes the examples it generates. So rather than generating really complicated examples as soon as it finds one that fails, it takes those examples and attempts to generate something simpler out of them.

I was thinking about this today and decided to have a play with putting something like this together in python. Rather than build a full testing setup I thought I’d instead just build the basic core algorithm and then figure out how to integrate this with py.test and similar later.

Enter hypothesis, a library for generating and minimizing data and using it to falsify hypotheses.

In [1]: from hypothesis import falsify
In [2]: falsify(lambda x,y,z: (x + y) + z == x + (y +z), float,float,float)
Out[2]: (1.0, 2.0507190744664223, -10.940188909437985)
In [3]: falsify(lambda x: sum(x) < 100, [int])
Out[3]: ([6, 29, 65],)
In [4]: falsify(lambda x: sum(x) < 100, [int,float])
Out[4]: ([18.0, 82],)
In [12]: falsify(lambda x: "a" not in x, str)
Out[12]: ('a',)

The current state of this is “working prototype”. It’s reasonably well tested, and the external interface to falsify is probably not going to change all that much, but the internals are currently terrible and liable to change almost entirely, but I’m reasonably happy with it as a proof of concept.

This entry was posted in Hypothesis, Uncategorized on by .

2 thoughts on “Falsifying hypotheses in python

  1. Omar

    Nice! It’s good to see a QuickCheck-inspired library in Python.

    Could you say a little bit about why you think Scala Check is nicer than Quick Check?

    1. david Post author

      There are basically three things I think scalacheck does better than quickcheck:

      • The minimization feature I mentioned: Scalacheck typically produces very nice counterexamples because it does quite a good job at minimizing the counterexamples. This is something I’ve tried to replicate with hypothesis: It doesn’t just generate counterexamples, it does its best to generate nice counterexamples
      • A lot more of it is first class values you can pass around. In particular you have much better control over building generators. Quickcheck’s reliance on type classes is in comparison very inflexible (this is actually a general advantage of Scala’s implicits over Haskell’s type classes, but Scalacheck uses it very well)
      • The stateful testing feature of ScalaCheck is unbelievably awesome for finding bugs in mutable data structures.

      I should also note that hypothesis isn’t exactly a quickcheck. The goal is that you could build quickcheck style stuff on top of it very easily (I intend to do so once I’ve got it in a state I’m a little happier with), but really it just encapsulates the core algorithms independently of any testing framework.

      In the mean time, if you want something that is a little more literally quickcheck for python, nat pryce has written factcheck, which is rather good from what I’ve seen of it. It doesn’t do any of the minimization stuff, but as a straightforward to use “Write tests based on random data” which works well with py.test it likely does exactly what you want.

Comments are closed.