My favourite language feature

I’ve been writing mostly Python and C for a while. This means that the features I’m used to are mostly Python ones, because C doesn’t really have any features so it’s hard to get used to.

And this means that when I write other languages I’m surprised because there are things that are ubiquitous in Python that I’ve forgotten aren’t ubiquitous everywhere. Most of them aren’t that important, but there was one that popped out at me that on thinking about I decided that I really liked and thought it was a shame that not every language implemented it.

That feature is named arguments with defaults.

It’s definitely not a Python specific feature – there are a bunch of other languages that have it – but Python is the language where I’ve really gotten used to how useful they are, and it’s far from ubiquitous.

I don’t really like Python’s implementation of it that much to be honest, but most of the flaws in it are easy enough to work around (and PEP 3102 makes life much better, or would if I didn’t have to write Python 2 compatible code), and when you do it makes it much easier to create extremely useful and flexible APIs. See my usage in hypothesis.strategies for example. e.g. the lists has 6 (ok, 5 really. The first one is only defaulted for legacy reasons) default options, all of which are useful and all of which play well together. Trying to do something similar with overloading or with different function names would be a complete horror, and really the best way to do it without default arguments is some sort of builder object which emulates them.

In my ideal world, I think this is how named arguments with defaults would work:

There is a hard separation between named and positional arguments. The names of your arguments are not significant unless you declare them as named arguments, and named arguments cannot be passed positionally. A function may take both positional and named arguments, there’s just a clear delineation between the two. This is basically essential if you want it to be possible to make forwards compatible APIs.
Named arguments are not required to have defaults.
Positional arguments cannot have defaults (I’m not heart set on this, but it seems like a feature that is very much of limited utility and it’s cleaner to not have it)
Defaults arguments are evaluated as expressions in the defining scope (not the calling scope) each time they are used. None of this ‘You can’t use [] as a default argument because it’s only evaluated once and then you’re sharing a mutable object’ nonsense from Python.
Default arguments may not depend on other named argument values. Sorry. I know this is useful, but it messes with evaluation order in the calling scope really hard and it just doesn’t make sense as a feature.
Optional: Default arguments may depend on positional argument values. This seems like an acceptable compromise for the preceding.

That’s pretty much all I have to say on the subject of named arguments with defaults: They’re great, more APIs should make extensive use of them where possible, and more languages should implement them.

Runners up

There are a bunch of other features that I think are great. Some of them made it on to my wish list but a lot didn’t for the same reason they didn’t claim top place for this post: They’ve already won. Support isn’t always ubiquitous, but it’s close enough that languages that don’t support them are weird and backwards outliers. Examples include:

Namespaces with renaming on import available (If you don’t have renaming on import then you don’t have namespaces, you have implementation hiding and are still shoving APIs into the global scope).
Local named and anonymous function definitions (“lambdas” or “closures”, but also nested functions).
Parametrized types and functions over them (for statically typed languages).

I’d probably take any of those over named functions with default, but fortunately i mostly don’t have to.

So who wants a Hypothesis port?

For those just joining us: Conjecture is a radical simplification of property based testing, as popularised by QuickCheck. It emerged as a simplification of my work on Hypothesis, which is probably the second most advanced property-based testing system (after Quviq‘s QuickCheck for Erlang, and Hypothesis has a number of innovations over their work as well), and certainly the most advanced open source one.

Conjecture has some moderate usability advantages over Hypothesis, but its real advantage is ease of implementation in a new language: Hypothesis is almost impossible to port due to its extreme implementation complexity, while Conjecture is both easier to port and also can easily be written as bindings to a C library.

I’ve been holding off on progressing Conjecture out of its current research prototype state while I try to figure out the whole making open source work profitable thing, which is why it hasn’t seen any progress in a few months.

Unfortunately it turns out I’m really bad at not being allowed to work on software projects I’m interested in. What ends up happening is that I just write them in my head instead.

…which means that I accidentally figured out all the remaining questions for how to make Conjecture work. Then I cracked and couldn’t resist putting code to computer. As a result, I’ve put together a new prototype in Python and it has better data quality and shrinking than Hypothesis does. I now essentially consider the design of Conjecture a solved problem, and the open questions of whether it works answered with a resounding yes.

Moreover, it didn’t take me very long to implement at all. It took me so little time to implement that I’m reconsidering whether it’s actually even worth writing Conjecture as a C library. It’s almost easier just to port it.

So, uh, would you like one? Because I can write you one. My current estimate is that I can probably do a pretty solid implementation in a new language in not more than a month of work.

To be concrete about what I am offering:

In almost any language of your choice, I am prepared to write something that is reasonably close to the Hypothesis core API (say, something akin to given, note + most of hypothesis.strategies). It will probably have some additional functionality, but the Hypothesis API is well proven in terms of its usefulness which makes it a good starting point.
I do not anticipate that this will take more than a month, and a pretty usable prototype should emerge after about a week.
I would charge you a day rate for my work. In exchange you would get a license to use but not redistribute all future released versions of Conjecture for that language without complying with the terms of the GPL (which it is otherwise released under).

Exact details are somewhat negotiable, but if you’re potentially interested, drop me a line.

Let Hypothesis making your choices for you

I had a moment of weakness this morning and did some feature development on Hypothesis despite promising not to. The result is Hypothesis 1.14.0.

This adds a bunch of interesting new strategies to the list. One I’d like to talk about in particular is the new choices strategy.

What does it do?ell, it gives you something that behaves like random.choice only under Hypothesis’s control and subject to minimization. This more or less solves the problem I had a long and complicated post about a while ago for picking elements from a list. You can now do something like:

from hypothesis import given, strategies as st
 
 
@given(st.lists(st.integers(), min_size=1), st.choices())
def test_deletion(values, choice):
    v = choice(values)
    values.remove(v)
    assert v not in values

Then running this will print something like:

_____________________________________________ test_deletion ______________________________________________
test_del.py:4: in test_deletion
    def test_deletion(values, choice):
src/hypothesis/core.py:583: in wrapped_test
    print_example=True, is_final=True
src/hypothesis/executors/executors.py:25: in default_executor
    return function()
src/hypothesis/core.py:365: in run
    return test(*args, **kwargs)
test_del.py:7: in test_deletion
    assert v not in values
E   assert 0 not in [0]
----------------------------------------------- Hypothesis -----------------------------------------------
Falsifying example: test_deletion(values=[0, 0], choice=choice)
Choice #1: 0
===================

Note that the choices are printed as they are made. This was one of the major obstacles to implementing something like this in the past: The lack of the ability to display the results from within the test. The new note API offers a solution to this.

Having another go at a vanity IRC channel

A while ago I experimented with a vanity IRC channel: #DRMacIver on Freenode. It kinda petered out, but I thought it might be nice to try again, so I’ve resurrected it.

So do join. If you’re not familiar with IRC, join anyway! Here’s a link to join via IRCCloud (don’t worry, it’s free).

Channel rules:

Hypothesis code of conduct is in effect.
There is no concept of off-topic discussion. You can talk about whatever you like. However note that I will be talking a lot about what I’m doing, posting links to my stuff, etc.

I’m basically treating it as an extended semi off the record commenting system for this blog and for other places I post things on the internet, so you’ll also get links to my fanfic, things I need help with, etc. I’ll probably set up some github hooks so most of my projects send commits in there.

A whirlwind tour of the Hypothesis build

I’m so used to it at this point that I occasionally forget that one of the innovative things about Hypothesis is its ludicrously complicated build set up. Some of it is almost certainly better than what is currently common practice, so I thought I’d do a quick post of some of the highlights.

Everything is run on Travis, but I don’t use travis’s python support. Instead I manage a set of Travis builds myself using pyenv’s installer code (I don’t actually use pyenv other than the installer). This is partly because I have OSX builders turned on (Thanks Travis!) but they don’t support Python, but it also ensures that I have up to date versions of every Python version.
I also have a rather less elaborate version of the build running on Appveyor to do some basic windows testing.
On Travis, the entire process is driven by a combination of tox and make.
I use travis caching quite heavily to ensure that managing my own python installs is fast – most builds do not need to install python because they have it installed from a previous run. The same is also true of building numpy wheels (though I’ve discovered recently I missed a bit).
I only run coverage on one build step, which runs on a single Python version (3.5, currently). This runs a fast subset of the tests for coverage then fails the build if it gets less than 100% coverage. This excludes my compatibility layer (compat.py) and a handful of other lines marked with pragma. The assumption is that if I’ve got coverage on 3.5 that’s probably telling me enough and isn’t worth the build time of trying on other Python versions (this probably does mean I have less than 100% coverage on Python 2, but so far this has never caused a problem).
Note: Hypothesis lives in a src/ directory so that it’s deliberately not on the path when running tox. This is the correct thing to do, because it means you’re definitely testing the installed version of the library. It also means you need some custom coverage config.
I have a custom script to enforce a standardized header for all my python files. The script is a bit hacky, but it works quite well.
I have a check-format build step. This applies a number of formatting operations, include pyformat, isort, and the aforementioned header script. It then runs git diff –exit-code to assert that these formatting options did not make any changes to the code. It also runs flake8 (this could reasonably be part of a different step)
For each dependency (optional in Hypothesis’s case) I have a separate tox step that tests each version of that dependency other than the latest (minor if I trust them, patch if I don’t) that I support. These all run on a single version of Python (3.5, currently). These tests also run against the latest version of the library in the per python builders. I don’t run non-latest versions against each python as an attempt to keep the combinatorics under control.
With this many tests of a randomized API that has a timeout built into it, it’s hard to avoid some flakiness – previously there were a few tests that would harmlessly fail on maybe one builder every couple of runs. This added up to a quite unreliable build. I’ve recently been using the flaky library to mitigate that – a handful of tests are decorated with @flaky to allow them to be rerun if they fail. This has been invaluable for getting a reliably green build.

The overall result is a bit ridiculous. My current travis build time is about two and a half hours depending on the time of day (Travis is slower during US working hours). The actual wait time is less than that because of parallel builders, but it’s still not short.

I think it’s mostly worth it though. The overall results give me an amazing amount of confidence in the code. Hypothesis definitely isn’t bug free (bug free code basically doesn’t exist outside of safety critical industries), but it’s generally regression free code and I tend to find more bugs than Hypothesis users do.