Author Archives: david

Contributors do not save time

(This is based off a small tweet storm from yesterday).

There’s this idea that what open source projects need to become sustainable is contributors – either from people working at companies who use the project, or from individuals.

It is entirely wrong.

First off: Contributors are great. I love contributors, and I am extremely grateful to anyone who has ever contributed to Hypothesis or any of my other open source projects. This post is not intended to discourage anyone from contributing.

But contributors are great because they increase capabilities, not because they decrease the effort required. Each contributor brings fresh eyes and experience to the project – they’ve seen something you haven’t, or know something you don’t.

Generally speaking a contribution is work you weren’t going to do. It might be work you were going to do later. If you’re really unlucky it’s work you’re currently in the process of doing. Often it’s work that you never wanted to do.

So regardless of what the nature of the contribution, it creates a sense of obligation to do more work: You have to deal with the contributor in order to support some work you weren’t going to do.

Often these dealings are pleasant. Many contributions are good, and most contributors are good. However it’s very rare that contributions are perfect unless they are also trivial. The vast majority of contributions that I can just say “Thanks!” and click merge on are things that fix typos. Most of the rest are ones that just fix a single bug. The rest need more than the couple of minutes work (not zero work, mind you) that it took to determine that it was such a contribution.

That work can take a variety of forms: You can click merge anyway and fix it yourself, you can click merge anyway and just deal with the consequences forever (I don’t recommend this one), you can talk the contributor through the process of fixing it themselves, or you can reject the contribution as not really something you want to do.

All of these are work. They’re sometimes a lot of work, and sometimes quite emotionally draining work. Telling someone no is hard. Teaching someone enough of the idiosyncracies of your project to help them contribute is also hard. Code review is hard.

And remember, all of this is at the bare minimum work on something that you weren’t previously going to do just yet, and may be work on something that you were never going to do.

Again, this is not a complaint. I am happy to put in that work, and I am happy to welcome new contributors.

But it is a description of the reality of the situation: Trying to fix the problems of unpaid labour in open source by adding contributors will never work, because it only creates more unpaid labour.

Against Virtue Environmentalism

I came up with the term “Virtue Environmentalism” recently and I think it’s a good one and will probably be using it more often.

This came up when talking to a friend (this really is a thing that happened to a friend, I promise, though some parts of it also apply to me) about a frustrating experience he’d had. Afterwards he vented at me about it for a bit and we had a good conversation on the subject.

The friend in question cares a lot about the environment. He’s mostly vegan and donates a lot of money to a variety of environmental charities and generally spends a fair bit of time stressing out about global warming.

But he also drives. Like, a lot. And I don’t mean a Tesla (I don’t know enough about cars to tell you about fuel efficiency, but it’s a conventional engine). Both short distance and long road trips. There’s no physical reason he has to drive – he’s in tolerably good shape and could definitely cycle a lot of the places he drives to if he wanted, but he really likes driving.

These aren’t inconsistent positions. I don’t think I was the one who convinced him of this, but he’s basically on board with the idea of donating instead of making personal interventions. He’s decided quite reasonably that his life is significantly better for all this driving that he’s willing to make the trade off, and he donates more than enough to be massively carbon negative despite it even without the veganism.

But someone he met at a party recently really took issue with that, basically calling him a hypocrite. I’m not sure how the subject came up, but it got quite heated.

Over the course of the conversation it emerged that the person in question was not vegetarian and did not donate anything to charity, but was very conscientious about taking public transport everywhere they couldn’t cycle, turning off all the lights, recycling everything, doing home composting, etc.

One of these people is making a big environmental difference. The other one is giving the person who is making a big environmental difference a hard time for not making a big enough difference.

(Note: This account has been somewhat fictionalized to protect the guilty)

I’m going to start describing this behaviour as virtue environmentalism.

The term comes from ethical theory. Approximately, we have consequentialist ethics and virtue ethics (it’s more complicated than that, but that’s the relevant subset here).

Consequentialist ethics says that ethical behaviour comes from acts which produce good outcomes, virtue ethics says that ethical behaviour comes from acts which exhibit virtues.

Similarly, consequentialist environmentalism says that environmentally friendly behaviour comes from acts which produce environmentally good consequences, while virtue environmentalism says it comes from acts which exhibit concern for the environment.

So, donating money to charity is consequentially good but mostly not a virtue – sure, you might as well do it, but it’s not real environmentalism.

My biases are clearly showing here. I largely subscribe to consequentialist ethics, but think virtue ethics has its place. There are good arguments that virtue ethics produces better consequential outcomes in many cases, and also that it produces better adjusted people. I’m not sure I buy these arguments, but it’s a valid life choice.

But virtue environmentalism is mostly bullshit.

Atmospheric carbon and other greenhouse gasses are amongst the most fungible types of harm out there. If I pump 100 tonnes of carbon into the atmosphere (a very high footprint) and extract 110 from it into some sort of long term storage (e.g. donating to prevent deforestation or plant new trees), then I’ve removed ten tonnes of carbon from the atmosphere and as a result I’ve done more good than someone who has only pumped 5 tonnes of carbon into the atmosphere (a very low footprint) but hasn’t removed any.

Virtue environmentalism largely results in three things:

Spending lots of time and effort on actions that make no practical difference at all but are highly visible.
Feeling good enough about yourself that you don’t perform the actions that would actually help.
Pissing off other people and making them care less about environmentalism overall.

The third is particularly important. If we want our descendants to not gently broil in the inevitable consequences of our own environmental waste, we need to get everyone to start to taking this seriously, and if you keep telling people that the only valid way to do environmental change is this sort of hair-shirt-wearing nonsense then the result will be that people do neither that nor the actually useful actions they would probably be quite happy to do.

If you want to do “environmentally friendly” things that don’t help much but make you feel better then sure, go for it. But stop expecting other people to do the same if you actually want to help the planet instead of just feeling good about yourself.

Fuzzing through multi-objective shrinking

This is an experiment I’ve been running for the last couple of days (on and off and with a bunch of tinkering). It was intended as a prototype for using glassbox in a next-gen version of Hypothesis, but it’s proven interesting in its own right.

The idea is a specific automated way of using a test case reducer as a fuzzer using branch instrumentation (I’m using afl‘s instrumentation via the afl-showmap command): For every branch we ever observe the program taking, we try to construct a minimal example that hits that branch.

This will tend to produce interesting examples because you throw away a lot of extraneous detail that isn’t required to hit that branch. This is is particularly true of “tolerant” parsers which try to recover from a lot of errors.

How it works

The core idea is that we take a normal test case reducer and repeatedly apply it in a way that automatically turns it into a multi-objective reducer.

Say we have a function, label, which takes a binary string and returns a set of labels. Labels can be anything, but in the case of using AFL’s instrumentation they’re essentially branches the program can take along with a rough count of how many times that branch was taken (essentially because the branches are hashes so some different branches may end up equated with each other).

We replace the labelling function with a side-effectful version of it which returns the original results but also updates a table which maps each label to its “best” example we’ve seen so far. We consider a string better than another if it is either shorter or the same length but sorts lexicographically before the other (when viewed as a sequence of unsigned 8 bit integers).

We then repeatedly iterate the following process: Pick a label, take the best example for that label, and reduce that test case with respect to the condition that it has that label (updating the other test cases with every call).

There are various heuristics you could use to pick a label. The ones I’ve tried are:

Pick one of the labels which currently has the best example
Pick one of the labels which currently has the worst example
Pick any label, uniformly at random

Uniformly at random seems to work the best: The others have a tendency to get stuck. In the case of ‘best’ there are a lot of small labels and it ends up spending a lot of time trying to shrink them all, not doing very interesting work in the process. In the case of ‘worst’ it tends to spend all its time trying to shrink very hard to shrink labels and not getting very far. Uniformly at random seems to consistently make progress and find interesting results.

Optimisations

There are a couple of extra useful things you can do to speed up the process.

The first is that every time label is called you can mark the string as known. Then when shrinking instead of shrinking by whether the string has the label, you shrink by whether the string is either the current best for the label or is unknown.

This works because if the string were simpler than the current best and already known, then the current best would already have been updated to that string.

This is the equivalent of caching the predicate for delta-debugging, but you don’t want to cache the label function because its outputs are complex values (they’re sets of labels, so there are \(2^n\) distinct values even after interning) so end up consuming a lot of memory if you cache them.

The second is that you can often tell when a label is going to be useless to shrink and skip it. There are two things you can do here:

If when you tried to shrink a label it made no changes, you can mark that label as ‘finished’. If another shrink later improves the label, you remove the finished mark. A finished label cannot be shrunk further and thus can be skipped.
By maintaining a counter that is updated every time a label is improved or added to the table, you can tell if an attempt to shrink did anything at all by checking the counter before and after. If it did nothing, you can mark the string as finished. Any labels whose current best string is finished can also be skipped.

This also gives a way of terminating the fuzz when there’s nothing left that’s discoverable: If every label is skippable, you’re done.

Observations

This seems to work quite well in practice. Starting from a relatively large initial example, it quickly increases the number of labels by about an order of magnitude (some of these are just difference in branch counts, as AFL counts not just whether the branch was hit but also a bucketed version of how many times).

It also works pretty well at finding bugs. I’ve been running it for about 48 hours total (a bit longer by clock time but I turned it off in the middle while I made some changes) and it’s found two bugs in a widely deployed file format parser that’s been stable for a couple of years (I’ve sent both to the author of the parser, and don’t want to say which one it is until I’ve got permission to do so. I don’t think either of them are security issues but hard to say for sure). One of them is confirmed novel, and I haven’t heard back about the other one yet. It found the first one after about 10 hours, but that appears to have been mostly luck – rerunning with a bunch of changes that otherwise improved the process hasn’t refound that bug yet.

Anecdotally, almost all of the examples produced are not valid instances of the format (i.e. the tool under test exits with a non-zero status code). This isn’t very surprising: The expectation is that it will give you just enough of the file to get you to the point you’re looking for and then throw away the rest, which is unlikely to get you a valid file unless the branch you’re looking for is taken after the file validity has already been determined.

Comparison with AFL

In some ways this is obviously quite similar to AFL, given that it uses the same instrumentation, but in other ways it’s quite different. My suspicion is that overall this approach will work better as an approach to providing a corpus to AFL than it will just on its own, but it’s surprisingly competitive even without that.

In particular it seems like it hits an order of magnitude increase in the number of seen labels much faster than I would expect AFL to. I think it helps that it’s using AFL’s instrumentation much more extensively than AFL itself actually does – AFL just uses the instrumentation for novelty detection, whileas this approach actually treats each label as a target in its own right and thus can take much more advantage of it.

The AFL algorithm is roughly just to repeatedly iterates the following:

Pick an example from the corpus and mutate it
If the mutated example exhibits any labels that we’ve not previously seen, add it to the corpus

It’s not really interested in the labels beyond novelty detection, and it doesn’t ever prune the corpus down to smaller examples like this does.

This approach also has a “self-optimizing” character that AFL lacks: Because AFL never replaces examples in its corpus, if you start with large examples you’re stuck with large examples forever. Because of this, AFL encourages you to start with very small, fast examples. This approach on the other hand will take whatever large examples you throw at it and will generally turn them into small examples.

To be clear: This isn’t a better approach than AFL. Maybe if it were highly optimized, tuned and refined it would become at least as good, but even then they would both have strengths and weaknesses compared to each other. But it’s not obviously a worse approach either, and even now it has some interesting advantages over the approach that AFL takes.

Doing less by doing more

My caffeine free goal structure probably seems weird to most people. That’s because it’s based on a simple motivational trick which may be pretty counter-intuitive, but has worked extremely well for me.

That trick is this: Never set out to do less of something. Instead, set out to do more of something that excludes the thing you want to do less of.

I do this primarily with Beeminder goals, but I think it’s more generally applicable.

The reason for this is that setting out to do less of something works very badly for me psychologically: It feels like a punishment, or like I have to ask for permission to do something I want to do. This is basically guaranteed to make me resent the goal and want to avoid the whole thing. Additionally, it turns whatever I’m supposed to be doing less of into a scarce resource, which makes me slightly obsess over it.

Goals to do more of something on the other hand don’t have these problems. Instead of being punished, I’m being rewarded, and the thing I’m wanting to do less of isn’t scarce any more – I can have as much of it as I want once I’ve completed this challenge!

Intellectually I know that these end up with exactly the same result, but I’ve found that despite that this makes such a difference in practice.

It’s usually quite easy to reframe things this way too. The “days without” works well for a lot of things, or you can pick some beneficial activity that is mutually exclusive with the thing you want to avoid (e.g. instead of a “spend less time sitting” goal you could have a “spend more time walking” goal).

This may be unique to me, but it seems consistent with other people’s experiences and the psychology of it doesn’t seem especially peculiar to me. So if you’re struggling to do less of something, I’d recommend giving this a try.

An experiment in breaking a caffeine addiction

I have a long running addiction to caffeine, to the tune of having probably averaged more than 300mg / day (usually in the form of a significant quantity of black coffee) for most of my adult life.

I would like to eliminate this, at least for a time. I don’t regard it as a problem per se, but due to reasons I want to establish a baseline for what life without caffeine is like.

I’ve tried this a number of times, through a process of going cold turkey, suffering, and then giving up again after about a month when the prospect of feeling like this forever became too much to bear. In reality I believe that after a 2 or 3 months the addiction should have fully worn off and I should feel “normal” (I’m aware that the official claim is 1-2 weeks, but I don’t believe it and the literature on caffeine addiction is so scarce that I feel entirely justified in this), but it sure doesn’t feel that way at the time.

Additionally, sometimes I just have… lets say bad days, and caffeine really helps me get through those.

So it’s pretty clear that the past strategy of just trying really hard is not going to work and it was time to try something different.

Well, I’ve been trying something different for the past few months, and while I’m definitely not out of the woods yet, I think I’ve found a pretty good path to getting there and I’m confident enough that this is going to work that I thought it might be time for a progress report.

It comes in two parts:

The first step was to decouple caffeine from the ritual of coffee. I usually started my mornings with a strong cup of black coffee, and I’d be the first to admit that this served a psychological purpose as much as a physiological one.

This turned out to actually be quite simple to do: Get a good decaf coffee and some caffeine pills. I’ve been drinking the Monmouth decaf and while it’s certainly not the best coffee I’ve ever had, it’s solidly in the category of good coffee. This means that I can start the day with a nice strong cup of coffee that happens not to contain caffeine, and control my caffeine intake entirely in pill form.

Drinking decaf coffee and taking caffeine pills seems perverse, but it’s actually been really useful. As well as the intended benefit of decoupling the two from each other so that I can have either of coffee or caffeine without the other, the caffeine pills make it much easier to regulate exactly how much caffeine I have. I suspect the fact that it makes it very obvious that this is a drug I’m taking doesn’t hurt either.

Anyway, that wasn’t enough to actually break the addiction, but I felt it did help a lot in getting a handle on my intake.

The second thing, which is by far the more important of the two, is a new experiment in using Beeminder.

The goal is to move to a habit where I spend increasingly long periods of time without caffeine. There’s no expectation that I’ll quit caffeine outright, just that I’m going to spend enough time off caffeine to keep the worst of the addiction at bay. Then, over time, I can gradually increase the length of these periods.

This does mean that I spend life in constant low grade caffeine withdrawal, but it’s very low grade, and it comes with the benefit of never having to deal with the really terrible caffeine withdrawal that comes from a full blown addiction.

The way the Beeminder usage works is to commit to having days without caffeine such that longer periods are rewarded more by gaining more “points” the longer I’ve been without caffeine. I score one point for the first day without caffeine, two for the second, three for the third, etc. I then have to make up a certain number of points per week.

This has been working pretty well. The system encourages me to keep up streaks without enforcing it – I get to make lots of tradeoffs of the form “Do I really want to have coffee today as much as N days worth of coffee?”, so unless I really need the coffee I often don’t have it.

I’m about to start an experiment where I also get negative points for days with coffee in the same way: -1 for the first day, -2 for the second, etc. This is because I’ve noticed that if I have about a week on coffee every day then coming off it gets unpleasant again.

You can see my progress here (and maybe predict how grumpy I’m likely to be on any given day based on that…) and the community policing thread here. I’ll report back at some later date how it’s all going.