As part of my drive to make Hypothesis more of a community project, one of the things I need to do is to ensure it’s easy for new people to pick up, and easy for people who have a different environment to use.
There are a couple major consistent sources of issues people have with Hypothesis development:
- It requires the availability of a lot of different versions of Python. I use pyenv heavily, so this hasn’t been a major problem for me, but other people don’t so are less likely to have, say, both Python 3.5 and Python 3.4 installed (because of reasons some build tasks require one, some the other).
- A full test run of Hypothesis takes a very long time. If you don’t parallelise it it’s in the region of 2 hours.
- Some of the build steps are very counterintuitive in their behaviour – e.g. “tox -e lint” runs a mix of linting and formatting operations and then errors if you have a git diff. This is perfectly reasonable behaviour for running on a CI, but there’s no separate way of getting the formatter to fix your code for you.
Part of the problem in 3 is that tox is a test runner, not a general task runner, and there was a lack of good unified interface to the different tasks that you might reasonably want to run.
So I’ve introduced a new unified system which provides a much better developer experience, gives a single interface to all of the normal Hypothesis development tasks, and automates a lot of the issues around managing different versions of Python. Better yet, it’s based on a program which is widely deployed on most developers’ machines, so there’s no bootstrapping issue.
I am, of course, talking about a Makefile.
No, this isn’t some sort of sick joke.
Make is actually pretty great for development automation: It runs shell commands, checks if things are up to date, and expresses dependencies well. It does have some weird syntactic quirks, and writing portable shell isn’t exactly straightforward, but as an end user it’s pretty great.
In particular because the makefile can handle installing all of the relevant pythons for you (I shell out to pyenv‘s build plugin for this but don’t otherwise use pyenv for this) the juggling many pythons problem goes away.
Other highlights:
- ‘make check-fast’ for running a fast subset of the tests
- ‘make format’ for reformatting your code to the Hypothesis style
- ‘make check-django’ and ‘make check-pytest’ for testing their respective integrations (there’s also ‘make check-nose’ for checking Hypothesis works under nose and I never giggle when typing that at all).
You can see the full Makefile here, and the CONTRIBUTING.rst documents some of the other common operations.
Here’s an asciinema of it in action:
I don’t know if it’d be helpful, but one method I’ve used in the past to support a wide range of Python versions in testing is documented here: http://ideas.offby1.net/posts/docker-runner-for-tox.html
Briefly, it involves a docker base image that knows how to run tox and provides the relevant interpreters pre-installed. It’s very linux-centric, but for application-level Python code I find that’s usually more than good enough. If it helps, cool!
Thanks, but I don’t really feel it’s worth the docker dependency. Getting people to run docker doesn’t seem easier than just downloading a bunch of pythons and sticking them on the path before running tox, and I’m not really clear what it buys you.
Also, I think a lot of people want to develop Hypothesis on OS X. I don’t, personally, but a lot of the point of this is making it easier for people who aren’t me.
This kind of bootstrapping dependency is something we’ve struggled with on Growstuff, one of whose aims is to be as easy as possible for non-coders to get started with. We’ve looked into Vagrant, Docker and so on, but installing them (especially on the low-end machines running random old OS versions that people typically come to us with) is barely any easier or less confusing than installing the dependencies directly, and anything that involves VMs sucks performance horribly.
FWIW, this approach 100% works for Ruby too. You have to faff around a little more with things like GEM_HOME and GEM_PATH environment variables, but rbenv comes with a good builder plugin you can use for installing and if you set those environment variables in the makefile everything just works.
Much though I hate Make itself, I thoroughly approve of this decision. It’s astonishing how many things that claim to be build tools punt on the central problems of dependency-graph management and checking up-to-dateness.
Pingback: A whirlwind tour of the Hypothesis build | David R. MacIver