Category Archives: programming

Free work is hard work

As has previously been established, I’m a big fan of paying people for labour and think you deserve to get what you pay for when it comes to software (yes, I do run a free operating system. And yes, I get what more or less precisely what I pay for).

But… honesty compels me to admit that paying for work is no panacea. If you compare open source software to similar commercial products, the commercial ones are usually really bad too.

They’re often bad in different ways, but they’re still bad. Clearly being paid for work is not sufficient to make good software.

I think I’ve figured out a piece of this puzzle recently: When working for free, the decision making process looks very different from paid work. The result is that you can be good at things that people aren’t good at when writing commercial software, but it’s also that you just do a lot more work that you would otherwise never have bothered with.

Consider the question: “Should I do this piece of work?”

Free answer: “Well… people seem to think it’s important, and it looks a bit shit that I haven’t done it. I guess I can find some free time to work on it.”

Paid answer: “Will I earn more money from the results of doing this work than the work costs me?”

The result is that in free work, a lot of things happen that if you put your hard nosed businessperson hat on and looked at objectively make absolutely no sense.

A concrete example: Hypothesis supports Windows. This isn’t very hard, but it does create an ongoing maintenance burden.

Personally, I care not one whit for Python on Windows. I am certain that near 100% of the usage of Hypothesis on Windows is by people who also don’t care about Windows but feel bad about not supporting it. I only implemented it because “real” libraries are cross platform and I felt bad about not running on Windows.

In a business setting, the sensible solution would be to not do the work until someone asked for it. and then either quote them for the work or charge a high enough license fee that you can soak the cost of platform support.

So what to do about this?

Well, on the open source side I think it makes sense to start being a bit more “on demand” with this stuff. I probably shouldn’t have supported Windows until I knew someone cared, and then I should have invited the person who cared to contribute time or money to it. Note: I am probably not going to follow my own advice here because I am an annoying perfectionist who is hurting the grade curve.

On the business side, I would advise that you get better software if you do sometimes do a bunch of stuff that doesn’t make immediate business sense as it rounds everything off and provides an over-all higher user experience even if any individual decision doesn’t make much sense. But honestly almost everyone writing software who is likely to read this is off in startup and adtech la la land where none of your decisions make any financial sense anyway, so that’s not very useful advice either.

So perhaps this isn’t the most actionable observation in the world, but now that I’ve noticed it’s going on I’ll be keeping a watchful eye out to observe its mechanisms and consequences.

My favourite language feature

I’ve been writing mostly Python and C for a while. This means that the features I’m used to are mostly Python ones, because C doesn’t really have any features so it’s hard to get used to.

And this means that when I write other languages I’m surprised because there are things that are ubiquitous in Python that I’ve forgotten aren’t ubiquitous everywhere. Most of them aren’t that important, but there was one that popped out at me that on thinking about I decided that I really liked and thought it was a shame that not every language implemented it.

That feature is named arguments with defaults.

It’s definitely not a Python specific feature – there are a bunch of other languages that have it – but Python is the language where I’ve really gotten used to how useful they are, and it’s far from ubiquitous.

I don’t really like Python’s implementation of it that much to be honest, but most of the flaws in it are easy enough to work around (and PEP 3102 makes life much better, or would if I didn’t have to write Python 2 compatible code), and when you do it makes it much easier to create extremely useful and flexible APIs. See my usage in hypothesis.strategies for example. e.g. the lists has 6 (ok, 5 really. The first one is only defaulted for legacy reasons) default options, all of which are useful and all of which play well together. Trying to do something similar with overloading or with different function names would be a complete horror, and really the best way to do it without default arguments is some sort of builder object which emulates them.

In my ideal world, I think this is how named arguments with defaults would work:

There is a hard separation between named and positional arguments. The names of your arguments are not significant unless you declare them as named arguments, and named arguments cannot be passed positionally. A function may take both positional and named arguments, there’s just a clear delineation between the two. This is basically essential if you want it to be possible to make forwards compatible APIs.
Named arguments are not required to have defaults.
Positional arguments cannot have defaults (I’m not heart set on this, but it seems like a feature that is very much of limited utility and it’s cleaner to not have it)
Defaults arguments are evaluated as expressions in the defining scope (not the calling scope) each time they are used. None of this ‘You can’t use [] as a default argument because it’s only evaluated once and then you’re sharing a mutable object’ nonsense from Python.
Default arguments may not depend on other named argument values. Sorry. I know this is useful, but it messes with evaluation order in the calling scope really hard and it just doesn’t make sense as a feature.
Optional: Default arguments may depend on positional argument values. This seems like an acceptable compromise for the preceding.

That’s pretty much all I have to say on the subject of named arguments with defaults: They’re great, more APIs should make extensive use of them where possible, and more languages should implement them.

Runners up

There are a bunch of other features that I think are great. Some of them made it on to my wish list but a lot didn’t for the same reason they didn’t claim top place for this post: They’ve already won. Support isn’t always ubiquitous, but it’s close enough that languages that don’t support them are weird and backwards outliers. Examples include:

Namespaces with renaming on import available (If you don’t have renaming on import then you don’t have namespaces, you have implementation hiding and are still shoving APIs into the global scope).
Local named and anonymous function definitions (“lambdas” or “closures”, but also nested functions).
Parametrized types and functions over them (for statically typed languages).

I’d probably take any of those over named functions with default, but fortunately i mostly don’t have to.

So who wants a Hypothesis port?

For those just joining us: Conjecture is a radical simplification of property based testing, as popularised by QuickCheck. It emerged as a simplification of my work on Hypothesis, which is probably the second most advanced property-based testing system (after Quviq‘s QuickCheck for Erlang, and Hypothesis has a number of innovations over their work as well), and certainly the most advanced open source one.

Conjecture has some moderate usability advantages over Hypothesis, but its real advantage is ease of implementation in a new language: Hypothesis is almost impossible to port due to its extreme implementation complexity, while Conjecture is both easier to port and also can easily be written as bindings to a C library.

I’ve been holding off on progressing Conjecture out of its current research prototype state while I try to figure out the whole making open source work profitable thing, which is why it hasn’t seen any progress in a few months.

Unfortunately it turns out I’m really bad at not being allowed to work on software projects I’m interested in. What ends up happening is that I just write them in my head instead.

…which means that I accidentally figured out all the remaining questions for how to make Conjecture work. Then I cracked and couldn’t resist putting code to computer. As a result, I’ve put together a new prototype in Python and it has better data quality and shrinking than Hypothesis does. I now essentially consider the design of Conjecture a solved problem, and the open questions of whether it works answered with a resounding yes.

Moreover, it didn’t take me very long to implement at all. It took me so little time to implement that I’m reconsidering whether it’s actually even worth writing Conjecture as a C library. It’s almost easier just to port it.

So, uh, would you like one? Because I can write you one. My current estimate is that I can probably do a pretty solid implementation in a new language in not more than a month of work.

To be concrete about what I am offering:

In almost any language of your choice, I am prepared to write something that is reasonably close to the Hypothesis core API (say, something akin to given, note + most of hypothesis.strategies). It will probably have some additional functionality, but the Hypothesis API is well proven in terms of its usefulness which makes it a good starting point.
I do not anticipate that this will take more than a month, and a pretty usable prototype should emerge after about a week.
I would charge you a day rate for my work. In exchange you would get a license to use but not redistribute all future released versions of Conjecture for that language without complying with the terms of the GPL (which it is otherwise released under).

Exact details are somewhat negotiable, but if you’re potentially interested, drop me a line.

New improved development experience for Hypothesis

As part of my drive to make Hypothesis more of a community project, one of the things I need to do is to ensure it’s easy for new people to pick up, and easy for people who have a different environment to use.

There are a couple major consistent sources of issues people have with Hypothesis development:

It requires the availability of a lot of different versions of Python. I use pyenv heavily, so this hasn’t been a major problem for me, but other people don’t so are less likely to have, say, both Python 3.5 and Python 3.4 installed (because of reasons some build tasks require one, some the other).
A full test run of Hypothesis takes a very long time. If you don’t parallelise it it’s in the region of 2 hours.
Some of the build steps are very counterintuitive in their behaviour – e.g. “tox -e lint” runs a mix of linting and formatting operations and then errors if you have a git diff. This is perfectly reasonable behaviour for running on a CI, but there’s no separate way of getting the formatter to fix your code for you.

Part of the problem in 3 is that tox is a test runner, not a general task runner, and there was a lack of good unified interface to the different tasks that you might reasonably want to run.

So I’ve introduced a new unified system which provides a much better developer experience, gives a single interface to all of the normal Hypothesis development tasks, and automates a lot of the issues around managing different versions of Python. Better yet, it’s based on a program which is widely deployed on most developers’ machines, so there’s no bootstrapping issue.

I am, of course, talking about a Makefile.

No, this isn’t some sort of sick joke.

Make is actually pretty great for development automation: It runs shell commands, checks if things are up to date, and expresses dependencies well. It does have some weird syntactic quirks, and writing portable shell isn’t exactly straightforward, but as an end user it’s pretty great.

In particular because the makefile can handle installing all of the relevant pythons for you (I shell out to pyenv‘s build plugin for this but don’t otherwise use pyenv for this) the juggling many pythons problem goes away.

Other highlights:

‘make check-fast’ for running a fast subset of the tests
‘make format’ for reformatting your code to the Hypothesis style
‘make check-django’ and ‘make check-pytest’ for testing their respective integrations (there’s also ‘make check-nose’ for checking Hypothesis works under nose and I never giggle when typing that at all).

You can see the full Makefile here, and the CONTRIBUTING.rst documents some of the other common operations.

Here’s an asciinema of it in action:

In praise of incremental approaches to software quality

Do you know what the best design decision I made in Hypothesis was?

It wasn’t well thought out. It wasn’t anything clever or theoretically interesting. It was done purely because I was lazy, not for any principled reason. It’s still the best design decision.

It’s this: Hypothesis does not have its own test runner. It just uses whichever one you’re already using.

This turns out to be a great idea, because it means that there’s basically zero cost to using Hypothesis: Install an extra library, write some code that uses @given to feed some data to a function. Done. Now you’re using Hypothesis.

As far as difficult sells go, this this barely even registers.

Compare this to, say, adding static analysis to your process (I’ve never got around to adding pylint to Hypothesis and I care about this stuff, because it gives too many false positives so it’s never been quite worth the up front investment), or rewriting everything in Haskell.

It’s certainly possible that both of those will produce enough of a return on investment from their correctness benefits that they’re worth the up front cost, but unless you’re really confident that they will and can afford to take the time to do it right now they’re things that you can put off basically indefinitely.

Hypothesis on the other hand is so cheap to get started with that you can start benefiting almost immediately, and then incremental amounts more work will produce incremental amounts more improvement.

This turns out to be incredibly valuable if what you actually want to produce better software.

One of the responses to the economics of software development post is that doing everything right up front can be cheaper than the long term cost of the unprincipled hacks that people often do instead.

This is certainly true. If you are lucky enough to find yourself in a position where you have the option of doing everything right up front, I recommend you make a reasonable effort towards doing so. You’ll probably fail (getting everything right is really rather hard), but in failing you will probably still end up in a better position than in not trying.

But, well, this is rarely the situation in which we find ourselves. Even if we got everything right at the beginning, at some point in the project we were rushed, or made a decision that later turned out to be a bad call, or in some other way failed to be perfect, because nobody can be perfect all the time.

If this is not the situation you find yourself in and you are working on a mature, high quality project that has consistently got most things right throughout its lifespan and has fixed the things it does not, good for you. Say hi to the unicorns for me while you’re there.

Most of us are not only not in that situation but that situation is almost entirely unreachable. Saying that if you did everything right at the beginning you wouldn’t have this problem is just another way of saying it sucks to be you.

So what we need are not tools that work great if you use them from the beginning, or tools that work great if you embrace them whole-heartedly, but tools that make it easy to get started and easy to move in the right direction.

This means that “rewrite it in Haskell” is out, but “we can write this microservice in Haskell” might be in. It means “Always have 100% branch coverage” is out, but “merges must never decrease branch coverage” is in. Failing the build if you have any static analysis warnings is out, but failing a merge if the touched lines have any static analysis failures is in.

Incremental quality improvements allow you to move your project to a higher quality state without requiring the huge up front investment that getting everything right from the beginning requires, and they don’t punish you for the fact that you didn’t get everything right before you had this tool to help you get things right.

This is of course often a harder problem, but I think it’s one that’s important to solve. If we want higher quality software we don’t get to live in a fantasy world where everything magically works, we have instead to think about how to move from the world we live in to a reachable world in which everything works better, and we need to think about what the steps along the way look like.