Version 0.16.0 of the Hypothesis pytest plugin is out. This contains two new features kindly contributed by others:
Effective immediately, there will be no further feature development from me on Hypothesis for the foreseeable future.
This is not the end of Hypothesis. I will still be:
Which is actually still quite a lot of work.
Hypothesis isn’t going anywhere. But, for now at least, 1.10 is as good as it’s going to get. This is now a side project, and you can expect a commensurate level of support. If you want more than that, or there’s some feature you desperately need, get in touch. We can talk rates.
At some point in the future I will doubtless resume feature development, but it won’t be soon.
Partly because Hypothesis 1.10 is good enough. It’s basically as good as or better than any open source equivalent in any language, and it’s certainly light years beyond what this time last year anyone in the Python community could reasonably expect would ever exist. There are exciting things I could work on, but they’d be another huge time sink and basically not all that helpful for the problem of making development on Hypothesis sustainable.
But mostly because I’m tired and I’m angry, and I’ve done a reality check and found reality wanting.
I’ve been trying to figure out a way of making Hypothesis development sustainable, and the answer is basically that I can’t, despite the fact that it’s clearly going to save people at the bare minimum millions of dollars over the course of its lifetime.
Yeah, I could probably eke out a living. Particularly if I was prepared to burn a lot of bridges and sacrifice most of what actually makes me want to work on it, but basically we’ve built an industry on free labour, and we’ve concluded that we’d much rather make people work for free in their spare time to produce adequate software and shame them into supporting it when somehow it surprisingly doesn’t do exactly what we want than fairly compensate for their labour and get good software out of it.
This makes any attempt to get money for tooling such an uphill struggle that it’s really not worth the effort. Plans which are predicated on changing the world before anyone will pay you any money are decidedly bad plans.
I think Hypothesis will make the world a better place, and I have a lot emotionally invested it, so as stated above I’m not abandoning it entirely, but I’ve really lost all desire to continue giving away so much of my labour for free, so I won’t.
There will be more on the subject of what happens next when I have collected my thoughts better and am slightly less angry.
In the short term though, if you’ve got any contracting work that you think would be up my alley and is either remote or reachable from London, that would be great.
The general consensus when I asked about this, which was conveniently what I was already planning to do, was to save Hypothesis 2.0 for a big release where I wanted to make some significant changes and was happy to cut the deprecated APIs and such at that point. So for now we’re continuing on with the 1.x series.
This release actually doesn’t have any new features externally. I’ve introduced a new ‘cleanup’ function, which lets you register cleanup handlers that execute at the end of your tests (this is more useful inside strategies than it is inside tests, but you can use it in either), but its semantics as it interacts with find are a bit weird and so I’m not currently declaring it part of the public API.
Other than that, this is mostly a bug fix and performance release. It was supposed to be a small release which just contained the performance improvements to one_of that I blogged about previously, but it got held up by a pypy bug (I believe this is the first time that’s happened – previous bugs have been easy to diagnose and work around) and then snowballed into a bit more significant of a release.
Also, reminder that I have a Patreon now, and that if you like this release schedule some support for it would be appreciated.
This is just a bugfix and performance release, but it changes some semi-public APIs, hence the minor version bump.
- Significant performance improvements for strategies which are one_of() many branches. In particular this included recursive() strategies. This should take the case where you use one recursive() strategy as the base strategy of another from unusably slow (tens of seconds per generated example) to reasonably fast.
- Better handling of just() and sampled_from() for values which have an incorrect __repr__ implementation that returns non-ASCII unicode on Python 2.
- Better performance for flatmap from changing the internal morpher API to be significantly less general purpose.
- Introduce a new semi-public BuildContext/cleanup API. This allows strategies to register cleanup activities that should run once the example is complete. Note that this will interact somewhat weirdly with find.
- Better simplification behaviour for streaming strategies.
- Don’t error on lambdas which use destructuring arguments in Python 2.
- Add some better reprs for a few strategies that were missing good ones.
- The Random instances provided by randoms() are now copyable.
- Slightly more debugging information about simplify when using a debug verbosity level.
- Support using given for functions with varargs, but not passing arguments to it as positional.
After a bunch of people have suggested it, I’ve created a Patreon to fund ongoing work on Hypothesis. Rewards include receiving in depth weekly updates about Hypothesis – news, advice on how to use it, etc. That probably means there will be a bit less of such things here.
I don’t think this will be enough to keep me in food and drink, let alone rent, and I still have ongoing plans for commercial development of things built on top of Hypothesis. This is purely to fund the development of the open source core library, and to give me enough money that I can feasibly keep going until I’ve got a more solid plan on that front.
So, if you like my work on Hypothesis (or even on here)! I’d really appreciate some funding. I’ve put a lot of work into this, and it sure would be nice to see something back for that.
I noticed about two hours after releasing that there’s a massive performance problem with the recursive data implementation I shipped in Hypothesis 1.9.0. Obviously this made me rather sad.
You have to do something slightly weird in order to hit this: Use recursive data as the base case for another recursive data implementation (actually using it in the expansion would probably work too).
The reason for this turns out to be nothing like what I expected and is kinda interesting, and I was really stuck as to how to solve it until I realised that with a little bit of lazy evaluation the problem was literally trivial to solve.
We’ll need to look at the implementation a bit to see what was happening and how to fix it.
Internally, Hypothesis’s recursive data implementation is just one big union of strategies. recursive(A, f) is basically just A | f(A) | f(A | f(A)) | … until you’ve added enough clauses that any subsequent one will basically never get in under the limit.
In order to understand why this is a problem, you need to understand a bit about Hypothesis generates data. It happens in two stages: The first is that we draw a parameter value at random, and the second is we pass that parameter value in to the strategy again and draw a value of the type we actually want (actually it’s more complicated than that too, but we’ll ignore that). This approach gives us higher quality data and lets us shape the distribution better.
Parameters can be literally any object at all. There are no valid operations on a parameter except to pass it back to the strategy you got it from.
So what does the parameter for a strategy of the form x | y | … look like?
Well, it looks like a weighting amongst the branches plus a parameter for each of the individual values. You pick a branch, then you feed the parameter you have for that branch to the underlying strategy.
Notably, drawing this parameter requires drawing a parameter from each of the underlying strategies. i.e. it’s O(n) in the number of branches.
Which means that if you have something like the recursive case above, you’re doing O(n) operations which are each themselves O(n), and you’re accidentally quadratic. Moreover it turns out that the constant factor on this may be really bad.
But there turns out to be an easy fix here: Almost all of those O(n^2) leaf parameters we’re producing are literally never used – you only ever need the parameter for the strategy you’re calling.
Which means we can fix this problem with lazy evaluation. Instead of storing a parameter for each branch, we store a deferred calculation that will produce a parameter on need. Then when we select a branch, we force that calculation to be evaluated (and save the result in case we need it again) and use that. If we never need a particular parameter, we never evaluate it.
And this means we’re still doing O(n) work when we’re drawing the parameter for a branch, but we’re only doing O(1) work per individual element of the branch until we actually need that value. In the recursion case we’re also saving work when we evaluate it. This greatly reduces the amount of work we have to do because it means that we’re now doing only as much work as we needed to do anyway to draw the template and more or less removes this case as a performance bottleneck. It’s still a little slower than I’d like, but it’s in the category of “Hypothesis is probably less likely to be the bottleneck than typical tests are” again.
In retrospect this is probably obvious – it falls into the category of “the fastest code is the code that doesn’t execute” – but it wasn’t obvious to me up front until I thought of it in the right way, so I thought I’d share in case this helps anyone else.