The idea of stateful testing is that it is “quickcheck for mutable data structures”.
The way it works is that rather than trying to produce arguments which falsify an example, we instead try and produce a sequence of operations which break a data structure.
Let me show you. We’re going to start with the following broken implementation of a set:
class BadSet: def __init__(self): self.data = [] def add(self, arg): self.data.append(arg) def remove(self, arg): for i in xrange(0, len(self.data)): if self.data[i] == arg: del self.data[i] break def contains(self, arg): return arg in self.data |
Because it uses an array internally to store its items and doesn’t check if an item is already contained when adding it, if you add an item twice and then remove it then the item will still be there.
(Obviously this is a really stupid example, but it should demonstrate how it works for more complicated examples)
Now lets write some tests that break this!
The operations we want to test are adding and removing elements. So we do the following:
from hypothesis.statefultesting import StatefulTest, step, requires class BadSetTester(StatefulTest): def __init__(self): self.target = BadSet() @step @requires(int) def add(self,i): self.target.add(i) assert self.target.contains(i) @step @requires(int) def remove(self,i): self.target.remove(i) assert not self.target.contains(i) |
We can now ask this to produce us a breaking example:
>>> print BadSetTester.breaking_example() [('add', 0), ('add', 0), ('remove', 0)] |
Note the nicely minimized example sequence.
At the moment this code is very much a work in progress – what I have works, but should probably be considered a sketch of how it might work rather than a finished product. As such, feedback would very definitely be appreciated.
Pingback: Another new hypothesis feature: flags | David R. MacIver
Looks interesting! Just curious, is the library code published in public somewhere?
Yes, the code is on github. There’s also a version of the library on pypi, but the stateful testing stuff hasn’t made it into a release yet.