Author Archives: david

The beginning of a novel that I’ll doubtless never write

Advance warning: This post is fiction. I had a fun idea and I needed to make word count, so I thought I’d write it up. However it’s not short fiction. It’s just a fragment of a larger story that will probably never get finished.

Other advance warning: Note that I’m not actually very good at writing fiction. In particular dialogue is not a skill I’ve ever been good at and some of the conversation in this piece really shows that. This is somewhere between an exposition dump and a first draft level of quality.

I’ve always wanted to be a wizard.

I think part of it is that I grew up in a library. My parents are scholars, and it always felt like this was basically like being wizards but worse. They had the silly robes, the books, the slightly distracted look, they just didn’t have the power.

I really wanted the power. I wanted to make a difference.

Which is why every year since I turned twelve I turned up for the testing.

If you’ve never been to a testing: They sit you down in front of a crystal ball, you have about a minute to stare at it and try to make it glow. If you’re wizard material it looks like staring at a bright flame. Most people can barely manage a glimmer.

I thought I managed a glimmer once, but it was just reflected sunlight. The rest of the time the crystal just sat there completely dead, taunting me.

Every time I hoped it would be different. Maybe it was just taking a long time to manifest? It does with some people. It was never different.

And then it was.

It was the last chance I had – after you’re 18 they won’t test you – but for once I wasn’t nervous. Some time, about six months ago, I woke up and the world looked different – sharper, brighter, and as if everything was suffused with a gentle glow. Since then the struggle has been to not cast spells. Kettles have boiled before I put them on the stove, the object I was searching for came flying to my hands, and countless other small incidents. I wanted something to happen, and then it happened.

When I sat down in front of the crystal I did feel a little nervous. I knew this time would be different, but what if it wasn’t?

I needn’t have worried. The crystal blazed like the sun. I was going to be a wizard.

The testing is only the beginning of course. After that they whisk you away to the college and put you through the wringer. They invade your mind – making sure you’re not a spy, that your intentions are good, and that you have a personality they think they can trust with magic. Most people make it through this stage, but a good quarter of people who pass testing just quietly disappear.

This stage had me rather more nervous. With good reason as it turned out, though in the end I made it through without a problem.

At the end of the day I had a set of apprentice robes, a room of my own (hardly more than space for a bed and a cupboard, but it sufficed) and the most complete sense of exhaustion I had ever experienced in my life.

I lay down on the bed and passed out almost immediately. But first, a single thought crossed my mind: Well that went about as well as we could possibly have hoped for.

And then I slept, and I remembered.


Six months ago I’d woken up to find a strange man in my room. Naturally, I shrieked. Who wouldn’t?

Then I noticed the robes and the rings. The man was obviously a wizard. My concern immediately turned to excitement. Was he here to tell me it had all been a mistake? That I had some sort of secret power the testing couldn’t have picked up and that he wanted to take me as his apprentice?

Then it turned back to concern. A lot of wizards are… not nice people, and the fact that the strange man who appeared in your bedroom in the middle of the night has power beyond your imagining and is basically untouchable by all earthly recourse is not actually a reassuring thought when you think about it.

“Don’t worry, Asha, I’m not here to hurt you”. He smiled in a way that someone who didn’t really know how to give a reassuring smile might do when they wanted to do so anyway. It did not entirely achieve its aims.

“I am Kelmir, and I’m here to offer you a deal. First I must know: Do you know how to accept a wizard’s oath?”

I nodded wordlessly. My parent had made sure of it – between my interest in magic and our proximity to the college they thought it was important I learned all the arcane etiquette.

“Very well”.

He took a ring off his finger, placed it in the palm of his hand and reached it out to me. I clasped his hand and felt the magic take hold.

“I am Kelmir, a mage of the collegium. As witnessed here, I swear by my life, my soul and my magic that all words I shall speak no lies to Asha, either by commission or omission. I shall tell him everything I think he needs to know and answer all his questions without reserve. If he does not agree to the terms I offer I shall wipe his memory of this entire event, but I shall not otherwise cast any spells upon him or offer harm to those he cares about without his complete and uncoerced consent. I am bound by these words”.

There was a glow around him as he said the final words, the ring grew hot in between our hands, and I felt a complete certainty that all of this was true.

If I was concerned before, now I was terrified. I’m sure a binding oath to cause no harm should be reassuring but I couldn’t imagine any circumstances which required something of that magnitude. I probably looked like I was about to pass out. Kelmir either didn’t notice or decided the best cure was to press on.

“First I must tell you one important thing: The wizard oath when offered to a non wizard may as well be a lie. If you had training and power it would be impossible to fake, but to someone such as yourself it would if anything be easier for me to create an illusion of the oath than to offer a true one. However I have actually offered you a true oath, which compels me to say this.”

I blinked.

“Uh. Is that supposed to make me trust you more or less?”

He grimaced. “To be honest I wasn’t planning to tell you that at all, but once the oath took hold I realised it would be a lie of omission not to tell you”.

That actually helped.

“The second thing I must tell you is that I believe you to be completely without magic and your dream of becoming a wizard is a futile one without my assistance. May I cast a spell on you to find out?”

Being told that felt like being punched in the gut. I had already known it was probably true, but I hadn’t wanted to admit it and resented being told. I nodded.

“I’m afraid you’re going to have to be more explicit than that”

I swallowed. “I give you my explicit consent to cast a spell on me to determine if I have the potential to be a wizard”.

He put his hand on my head and I briefly saw double. He withdrew it, and the world returned to normal.

“It’s as I thought. No magic at all. As things stand, you can never be a wizard”

“As… as things stand?”

“I’m here to offer you a way, but it comes at a price”
“I accept”

He grimaced.

“No you don’t. I need you to understand the way the world works first. You have to understand what it is I’m trying to do here, and you have to agree to my goals with that full understanding. This can’t work any other way”

“I don’t care what you’re asking for. I’ve always wanted to be a Wizard. I’ll do anything”

He gave me a searching look and nodded decisively.

“Alright then”.

My heart leapt.

“The first thing we have to do is kill your parents”

I froze.

“I… uh. I guess I was wrong. I don’t want to be a wizard that much”.

“Good. This would have been a very short conversation if you had. Now stop being an idiot and listen to what I have to say before you agree to anything.”

I put on my best contrite expression.

“Now. How many wizards do you think there are in the college?”

“I don’t know. Maybe a few hundred?”

“Yes, that’s about right. Now, why are there so few?”

“Well, hardly anyone has magic. Less than one in a hundred I guess? And there aren’t more than twenty thousand people near here, so that sounds about right?”

“And why are there only twenty thousand people near here?”

“Well, how could we feed much more than that? We get famines occasionally as it is! Also the last plague killed a lot of us. I think maybe there were more like 30,000 back then?”

“And why are there plagues and famines?”

I was getting increasingly confused by this line of questioning.

“I don’t really know? The priests say the gods send them to punish us, my parents say it’s just the natural order of things”

“And do you know of anyone who is able to change the natural order of things?”

That gave me pause. Of course I’d fantasised about making things better when I was a wizard, but somehow I’d never made the connection that there were all these wizards already and they weren’t actually doing anything to make things better.

“OK. I get it. So, tell me. Why don’t wizards prevent the famines and the plagues?”

“It’s really very simple: We don’t prevent the famines and the plagues because we want there to be famines and plagues”.

I’m not sure how long I sat there completely poleaxed, but eventually he figured I wasn’t going to be able to ask an intelligible question and continued.

“As far as the college is concerned, the status quo is pretty near perfect. We have a stable world in which we’re at the top. Why should we try to change that? If there were more people, the balance of power might shift, and if there were more wizards then each individual wizard would be less powerful because there would be more of us to split the power amongst. So we keep the world as it is: Every generation we get more powerful and the rest of you stay the same.”

“That’s… that’s monstrous”

“I quite agree. Which is why I want to you to help me change it”

“What can I do? You just said I have no magic”

“A wizard can pass on their magic before they die. Normally this is spread amongst a small number of their successors, which is how we accumulate power. I propose to pass it on to you instead.”

“Why me? What’s the catch?”

“Why you is because you are idealistic enough that I believe you will share my goals, and desperate enough that I believe you will accept the, as you put it, catch. My power will come alone: I will also be giving you my mind. At first I will be there as a voice in your head, but over time I think it likely that we will start to share thoughts and memories, and eventually we will become one person.”

“So I won’t be me any more?”

“Think of it as just a more extreme form of how you’ll change over time anyway. I think we are compatible enough that you won’t find the changes too objectionable”.

We talked for a lot longer than that, but the outcome was never in doubt. I accepted his life, and his magic, and then he showed me how to bury the memories so deeply that the tests at the school would never find them.

I woke up the morning after the tests feeling refreshed. The memory blocks were gone, and I knew exactly where we were and who we were.

Do you remember the plan? Kelmir asked me.

Oh yes. I remembered the plan rather well. Play the part of a powerful, intelligent, but very inexperienced student. Make alliances with anyone who could be persuaded that change was essential, or at least persuaded to acquiesce to it. Deal with the rest. If necessarily, permanently.


So that’s it for now. As mentioned I will probably never actually write more on this. We’ll see.

As you can probably tell this is a thinly veiled analogue about entrenched capital, and it doesn’t even attempt to veil that it’s about how shit being in a feudal society is, it’s just that there are wizards instead of lords.

Essentially this is mostly a reaction to the fact that I enjoy wizard school books but find their politics almost uniformly terrible.


This entry was posted in Uncategorized on by .

How to stop Hipchat beeping at you.

Due to reasons, I’ve been hanging out in a Hipchat channel as a guest recently.

It’s not the greatest of software in the world, but it’s mostly not terrible. Except for in one respect.

Someone suggested two years ago that maybe us guests would like to be able to turn off the intensely annoying beep that you get on every message (I really really hate software that makes noises at me). So far Hipchat have not bothered to implement this.

I’d previously been using the following javascript in the console to pick up the slack:


For some reason this has stopped working today. I don’t know why. Some further poking around in the internals has given me the following better invocation:


This seems to still work.

I’d make some disparaging comment about having to work around other developers’ mistakes at this point, but with UX this bad it feels practically redundant.

This entry was posted in Uncategorized on by .

Triangular Beeminding; Or, Drink Less, Using the Power of Triangles

This is a crosspost of a guest post I’ve written for the Beeminder blog.

One of my vices is that I drink a bit too much. Not to the level where I have a problem, but it would be strictly better if I cut out about 2 or 3 of the drinks I have in a typical week. This seems like an obvious use case for Beeminder.

I’ve previously beeminded units of alcohol consumption and concluded that, measured as a total number of units per week, I’m completely fine. The recommended maximum intake for an adult male is somewhere in the region of 20 – 25 units per week depending on who you ask. When I was beeminding this regularly I never had trouble keeping under 12 units. I drink a bit more than that now, but nowhere close to twice as much.

So if I’m that far under the recommended guideline why do I think I drink too much?

Well, the low average is because I actually have a lot of nights of any given week where I don’t drink at all. The problem is that on nights that I do drink I often have a drink or two more than I should. I make tasty cocktails, and if I’ve just had a cocktail I really liked then making another one sounds like an excellent idea. By the third drink of the evening I will usually discover the next day that it wasn’t such an excellent idea.

So what I need is a Beeminder goal that matches the structure of the behaviour I want to change: I need a way to beemind the peaks as well as the averages. A week with two three-drink nights should “cost” more than a week with a single drink every night.

I’ve come up with what seems like a good structure for this.

The idea is to assign each drink [1] a number of points. The first drink in a day costs one point, the second two, the third three, and so on. Because these add up, this means that a day with one drink costs one point, a day with two drinks costs three, a day with three drinks costs six. It mounts up pretty quickly. These running totals are called triangle numbers, hence the title of this post.

To start with I’ve capped the total number of points at 15/week. This is a deliberately lax starting rate which equates to a maximum of 11 drinks in a week (3 days with 1 drink and 4 with 2). Since the drinks I tend to have are two units this is about at the recommended maximum. Note that I can hit the limit while drinking less than that: If I have more than two drinks on any night, the extra points mean I’m forced [2] to reduce the total for the rest of the week to compensate.

Example permitted maximum drinking patterns:

  1. 4 days with 2 drinks and 3 days with 1 (11 drinks)
  2. 1 day with 3 drinks, 1 day with 2 drinks, 5 with 1 (10 drinks)
  3. 2 days with 3 drinks, 3 days with 1 drink, 2 days alcohol free (9 drinks)
  4. 1 day with 4 drinks, 1 day with 2 drinks, 2 days with 1 drink (9 drinks)
  5. 1 day with 5 drinks (!) and no drinking the rest of the week (5 drinks)

Note that 3 days with 3 drinks is not permitted even with the remaining rest of the week free: That would be 18 points which would take me over the threshold [3].

I was originally planning to track this manually, but then Danny got so excited by the concept that he added a feature for it, so it’s easy to give this a try yourself:

  1. Go to “Terrifyingly advanced settings”
  2. Convert your goal to a custom goal (this requires a premium plan).
  3. Switch the aggregation mode to “Triangle”

Best to apply this to a fresh goal. This stuff can easily screw up your goal if you’re not careful, so don’t do it to one with data you care about. [4]

If you want to track it manually instead, just enter the numbers yourself: 1 for the first drink, 2 for the second, and so on [5]. A standard Do Less goal will sum those up, yielding the triangular numbers.

So far this is experimental. I’ve only been running this for a few days, so it may turn out to be a silly idea in the long run. I don’t think it will though. I’m quite pleased with the incentive structure it sets up, and the effect so far has definitely been to make me think more carefully about the later drinks. I’ll add a follow up comment to this post in a month or so when I’ve had time to see how it works.



[1] Drinks, not units of alcohol. I try to keep my Beeminder goals based on things I don’t need to estimate or measure. Especially if I have to estimate them after a few drinks. Most of my drinks are approximately two units as I tend to drink cocktails or spirits.

[2] Well, “forced”. I have this goal set up so that it’s OK to fail occasionally. I’ve got a pledge cap of $10 set, so the worst case scenario is that my drinks suddenly become a bit more expensive. This is coupled with a no-mercy recommit: If I decided last night that I was OK derailing, I’m not off the hook today. This is based on a concept from Bethany Griswold about using Beeminder to make free things not free. The Bethany better known in these parts makes a related point in “Be Nice To Yourself”.

[3] Normally this wouldn’t be quite true because I could build up buffer from week to week if I wasn’t drinking much, but I’ve got this goal set to auto-ratchet so I can’t actually do that. If I build up more than a week of buffer it cuts back down to a week. Another terrifyingly advanced premium feature, available with Plan Bee.

[4] Danny here: But we’re excited about people trying this so actually please do feel free to experiment and holler at [email protected] if you break something and we’ll fix it!

[5] Danny doesn’t like this because it breaks the “Quantified Self First” principle. The numbers that you enter this way don’t correspond directly to something you want to measure. [6] Personally I’m much more interested in behaviour change than QS, so I don’t have a problem with it.

[6] Danny again: Actually, QS is more about measuring real-world things. With the triangle aggregation, we’ve got the best of both worlds. We get a true count of the number of drinks (real-world measure) and we get a goal-friendly aggregation for beeminding. You can export the data and do something else with it or change the aggregation function back to normal summing if you want to go back to beeminding total amount of alcohol consumed.

This entry was posted in Uncategorized on by .

Monadic data generation strategies and why you should care

I posted this gist earlier. It’s a toy port of one of the new templatized data generation for Hypothesis to Haskell.

It doesn’t do most of the important things. Its purpose is to demonstrate one simple point: Hypothesis strategies are now monads.

I don’t want to get into the deep philosophical implications of this or how it means Hypothesis is now like a burrito in a space suit. What I want to do here is point out that this enables some super useful things.

Consider the following example from the Hypothesis README (at the time of this writing. This is going to change soon for various reasons, oe of them being the stuff I’m about to get into)

from decimal import Decimal
from hypothesis.searchstrategy import MappedSearchStrategy
class DecimalStrategy(MappedSearchStrategy):
    def pack(self, x):
        return Decimal(x) / 100
    def unpack(self, x):
        return int(x * 100)

This is for defining a Decimal strategy in terms of an integer strategy – it has an operation to convert a decimal to an int and an int to a decimal.

The reason it needs to convert a decimal to an int is because of simplification. If it can’t convert back then it can’t simplify. This is also what stops strategies from being a functor (remember that all monads are functors): In order for something to be a functor we need to be able to define a new version by just mapping values. We can’t require the mapping to go both ways.

Which is why it’s pretty great that the new template API lets us throw away half of this! Now MappedSearchStrategy no longer requires you to implement unpack. As well as being half the work, this means you can use it in cases where you might sometimes need to throw away data – the mapping no longer has to be a bijection. The reason it can do this is that it just uses the templates for the original type, so there’s no need for you to convert back.

But that’s just an example where being a functor is useful. Why is being a monad useful?

Well, the operation that monads add over functors is bind. map let us take a function a -> b and turn a Strategy a into a Strategy b. bind lets us take something that maps a to a Strategy b and turn a Strategy a into a Strategy b. When would we want to do this?

Well, one example is when we want some sort of sharing constraint. Suppose for example we wanted to generate a list of dates, but we wanted them to all be in the same time zone. The bind operation would let us do this: We could do something like strategy(timezones).bind(lambda s: [dates_from(s)]) (this is a made up API, the details for this are not yet in place in actual Hypothesis). This would generate a timezone, then we generate a strategy for generating dates in that time zone, and a strategy for producing lists from that.

Given that this is Python, you don’t have the advantage you get in Haskell that being a monad gives you nice syntax and a rich set of support libraries, but that’s OK. The reason monads are a thing in the first place is that the monad operations are generally super useful in their own right, and that remains true in Python too.

This entry was posted in Uncategorized on by .

The three stage value pipeline for Hypothesis generation

Note: This describes a work in progress rather than something that is in a released Hypothesis version. As such it’s liable to change quite a lot before it’s done. I’m partly writing this as a form of thinking through the design, partly as a way of procrastinating from the fact that I’ve literally broken the entire test suite in the course of moving to this and trying to fix it is really making me wish I’d written Hypothesis in Haskell.

I’ve previously talked about how generating data in Hypothesis is a two step process: Generate a random parameter, then for a given parameter value generate a random value.

I’ve introduced a third step to the process because I thought that wasn’t complicated enough. The process now goes:

  1. Generate a parameter
  2. From a parameter generate a template
  3. Reify that template into a value.


The idea is that rather than working with values which might have state we instead always work with merely the information that could be used to construct the values. The specific impetus for this change was the Django integration, but it also allows me to unify two features of Hypothesis that… honestly it would never have even occurred to me to unify.

In order to motivate this, allow me to present some examples:

  1. When generating say, a list of integers, we want to be able to pass it to user provided functions without worrying about it being mutated. How do we do this?
  2. If we have a strategy that produces things for one_of((integers_in_range(1, 10), (9, 20))) and we have found a counter-example of 10, how do we determine which strategy to pass it to for simplification?
  3. How does one simplify Django models we’ve previously generated once we’ve rolled back the database?

Previously my answers to these would have been respectively:

  1. We copy it
  2. We have a method which says whether a strategy could have produced a value and arbitrarily pick one that could have produced it
  3. Oh god I don’t know can I have testmachine back?

But the templates provide a solution to all three problems! The new answers are:

  1. Each time we reify the template it produces a fresh list
  2. A template for a one_of strategy encodes which strategy the value came from and we use that one
  3. We instantiate the template for the Django model outside the transaction and reify it inside the transaction.

A key point in order for part 3 to work is that simplification happens on templates, not values. In general most things that would previously have happened to values now happen to templates. The only point at which we actually need values is when we actually want to run the test.

As I said, this is still all a bit in pieces on the floor, but I think it’s a big improvement in the design. I just wish I’d thought of it earlier so I didn’t have to fix all this code that’s now broken by the change.

(Users who do not have their own SearchStrategy implementations will be mostly unaffected. Users who do have their own SearchStrategy implementations will probably suffer quite a lot in this next release. Sorry. That’s why it says 0.x in the version)

This entry was posted in Uncategorized on by .