Category Archives: life

There are no hidden rules

Have you ever experienced the feeling that the people around you all know a bunch of secret rules? Moreover, they expect that you know them too and will punish you for not following them, but still won’t ever actually tell you what the rules are? Or sometimes they do tell you what the rules are but it later turns out that they were lying or they forgot to tell you a huge list of exceptions and just assumed that you understood those.

Yeah, me too.

(If you haven’t experienced this feeling, this post is probably not for you. You’re of course welcome to keep reading it anyway if you wish, but you might find it a bit obvious).

A friend was asking about how to figure out what these rules are the other day, specifically in the context of the workplace. I gave an answer, but the more I thought about my answer the more I realised that I hadn’t ever really internalised it or thought through the broader implications.

The answer is this: There are no hidden rules. This entire feeling is an illusion.

The feeling of confusion is real, and is caused by a real problem, but the problem can not be accurately described as there being “hidden rules”.

Why? First, we’ll need a brief digression on the nature of rules.

Rules mostly come in two forms: prescriptive and descriptive. A prescriptive rule is explicit and enforced. A descriptive rule just describes what is observed to happen.

Or to put it another way, when people are observed to violate a prescriptive rule, the people violating it are at fault. When people violate a descriptive rule, the rule is at fault.

When we’re worried about hidden rules, we’re worrying about prescriptive ones – we think that there is some secret rule book that we are bad for not following. It is more or less impossible that this could be the case.

In order for such a rule book to exist, people would have to be getting together behind our backs and conspiring to come up with and enforce these rules, and then not telling us about them. I am reasonably sure that this is not happening.

There is still a rough shared consensus, but it arises more organically than that. People come up with their own ideas of how things work by observing others and talking to them. Sometimes they have explicit conversations about how things are actually supposed to work, but it’s relatively rare, and over time a rough sort of knowledge builds up about how people are supposed to behave. But it is at best a rough consensus – different people start with different conceptions of it, and these mutate and change as they move from person to person.

The result is that everyone is operating with a different – sometimes subtly different, sometimes wildly different – set of assumptions about how people should behave, and the resulting mess does not resemble any sort of unified prescriptive set of rules that people follow. Instead of a single rule book, there’s a vague amorphous mass of roughly acceptable behaviours that contains a mix of vague consensus and massive individual divergence.

So if we want some rules that tell us how to behave, we’re left with descriptive rules – we need to somehow take all of the complex and varied space of human behaviour and boil down aspects of it to a simple set of formulae.

I don’t know if anyone’s mentioned this before, but it turns out that “solving literally the entire domain of social science” is a hard problem.

So there is no rule book. How do people even function?

Well, they mostly just get on with things. They do some complex mix of what they want, what they feel they should, and what they think they can get away with. They reward behaviours that they like, they punish behaviours that they don’t like.

The result is a system where the range of permitted behaviour is more a function of the people you’re interacting with, how they interact with each other, their individual wants and preferences, and a massive amount of context.

Is something permitted? Well, it depends. Who doesn’t want you to do it, who does want you to do it, and what are the relevant balances of power between those people? Or are you individually powerful enough that you just don’t really have to care in this instance and can do it anyway?

And this mostly works. People generally want to get along, so the group roughly stabilises itself into a functioning pattern that is more or less compatible with everyone’s behaviour, and self-corrects when someone goes too far out of line.

There will probably end up being some broad areas of acceptable and unacceptable ranges of behaviour – interactions that have been done often enough that they’re largely understood and people know how to act in them – but if you ask people what they are, they’re still going to have to figure out “the rule” based on observation: It is still a task of description.

Note that this is true even when people think they already know the rule. They just did it earlier.

People are good at not noticing the special cases to the things they think are true, so if you ask someone for the rule and they tell you, you’ll often spot exceptions. If you ask about those exceptions you’ll usually hear “Oh but that was a special case/justified because of this reason”. Often that reason also has exceptions. It is very unlikely that the person you are asking consciously “knew” all these exceptions before you asked, they just took them in stride as they happened and didn’t notice that they contradicted what they thought was true.

And yet, it sure seems like they have some sort of special insight that we lack, doesn’t it?

Well, they do, but that insight isn’t a rule book, it’s more awareness of the social context and better intuition about how other people are going to behave, usually based on the idea that others are going to behave in a broadly similar manner to how they would. If you’re coming into an unfamiliar cultural context (e.g. due to transition or immigration, or even just joining a company with a very different business culture) there is probably also a lot of shared history that you like, and assumptions that have built up over time from that.

But social science is still hard, and better data and intuition aren’t enough to change that, so that doesn’t actually mean they’re going to be right, they just might be a bit more accurate than we are.

Or they might be less accurate. Often when people try to make the implicit explicit they end up whitewashing reality – they give the nice version of the rule that they want to be true, which ignores a lot of the messy special cases they feel bad about. e.g. people will tend to tell you that lying is bad and of course you should be honest. Except for those little white lies and special cases that are required for social niceties. And naturally you exaggerate your performance to look good because everyone else is. Oh and when people say this what they really mean is… etc.

Even if they believe the rule is that you should always tell the truth, the practice does not follow and the more you tease them out on it the more they admit that the “real” rule has endless special cases and caveats and exceptions and actually once you consider all of the things that “don’t count”, people are lying to each other all the time.

The point is that “hidden rules that everybody knows” is not a useful mental model of what people are doing, because they don’t have access to any hidden rules either. They don’t have any sort of general theory or specific knowledge of what’s going on, and as a result any attempt to get them to tell you what the rule is will be futile.

Does this all sounds very bleak and unhelpful to you? It does to me, but I don’t actually think it is.

I think instead it’s bleak and helpful.

The most helpful thing about it is that you can just stop looking for the true hidden rules. It’s a waste of time. There aren’t any.

That isn’t to say rules can’t still be useful, just that it’s important to understand their nature before applying them.

You can think in terms of rules for you if that helps – things that make it easier for you to navigate the social situation if you follow them. As long as you understand that these are yours and under your control, and don’t expect other people to follow them, this can be very helpful.

You can try to come up with descriptive rules for how people will behave, but don’t sink too much time into making them accurate – it’s better to have an easy rule that you understand isn’t perfect than a slightly more accurate complex one that is much harder to work with and still isn’t perfect.

Often these will be quite different from the rules that people tell you if you ask. It can still be worth asking – the answers are useful data and might be worth a try – but you shouldn’t treat the answers as automatically correct.

These can be particularly useful when coming into a new context. When you’re in an unfamiliar situation, having these sorts of approximations to the shared cultural assumptions can be extremely useful for getting a basic handle on the expected behaviour until you manage to acquire a more intuitive sense of this. If you can find other people who have had a similar context switch to you, talking to them and asking what they think the rules are is also often more helpful.

This isn’t the scenario I usually find myself in, so my advice here is necessarily a bit vague. For me the problem, especially historically, is more often that I feel this way in scenarios that I have plenty of cultural context for but remain more than a bit mysterious anyway. I think this is a pretty typical experience for people with autism or who otherwise have minds and personalities sufficiently different from the norm that predicting that other people will behave more or less like you doesn’t really work.

And rules will only get you so far here. The thing to look at when you really want to understand how to navigate a group isn’t rules, it’s people. What they want, how they interact, and how those desires reinforce and conflict with each other.

This is unfortunate, in that people are much harder to understand than rules, but it does have the advantage that it might actually work.

This entry was posted in life on by .

Looking into doing a PhD

As regular readers of this blog have probably figured out, I’m a researchy sort of person.

A lot of my hobbies – maths, voting theory, weird corners of programming, etc – are research oriented, and most of my work has had some sort of research slant to it.

The last two years I’ve basically been engaged in a research project working on Hypothesis. It’s come quite far in that time, and I feel reasonably comfortable saying that it’s the best open source property based testing library on most metrics you’d care to choose. It has a number of novel features and implementation details that advance the state of the art.

It’s been pretty great working on Hypothesis like this, but it’s also been incredibly frustrating.

The big problem is that I do not have an academic background. I have a masters in mathematics (more technically I have a BA, an MA, and a CASM. Cambridge is weird. It’s entirely equivalent to a masters in mathematics though), but that’s where I stopped. Although it says “DR” in my online handle and the domain of this blog, those are just my initials and not my qualification.

As a result, I have little to no formal training or experience in doing academic research, and a similarly low understanding of who’s who and what’s what within the relevant fields. So I’ve been reading papers and trying to figure out the right people to talk to all on my own, and while it’s gone OK it’s still felt like fumbling around in the dark.

Which leads to the obvious solution that I spoilered in the title: If the problem is that I’m trying to do research outside of an academic context, the solution is to do research in an academic context.

So I’d like to do a PhD that is either about Hypothesis, or about something close enough to Hypothesis that each can benefit from the other.

There’s probably enough novel work in Hypothesis already that I could “just” clean it up, factor it out, and turn it into a PhD thesis as it is, but I’m not really expecting to do that (though I’d like that to be part of it). There are a number of additional directions that I think it would be worth exploring, and I expect most PhD funding will come with a focus subject attached which I would be happy to adapt to (a lot of the most interesting innovations in Hypothesis came because some external factor forced me to think about things in ways I wouldn’t otherwise have!). If you’d like to know more, I’ve written up a fairly long article about Hypothesis and why I think it’s interesting research on the main Hypothesis site.

Which, finally, brings me to the main point of the post: What I want from you.

I’m already looking into and approaching potential universities and interesting researchers there who might be good supervisors or able to recommend people who are. I’ve been in touch with a couple (some of whom might be reading this post. Hi), but I would also massively appreciate suggestions and introductions.

So, if you work in relevant areas or know of people who do and think it would be useful for me to talk to, please drop me an email at [email protected]. Or just leave a comment on this blog post, tweet at me, etc.

This entry was posted in Hypothesis, life, programming, Python on by .

2017 Goal: Recurring Revenue

It’s a bit early for new years resolutions, but I don’t believe in new years resolutions anyway so oh well.

I’ve been self-employed for nearly two years now, and an ongoing question is whether I should stay that way. I keep moving the goal posts for when it’s time to quit and get a day job, so this post is my attempt to fix a set of goal posts (or, well, to fix the derivative of the goal post’s position over time. More on that in a bit), as setting concrete decision criteria is the best way to deal with the various biases my brain throws out.

First, cards on the table: The reason this is a question at all is that from a financial point of view my attempts to go it alone are so far a failure. I am currently earning significantly less than I did in my first year as a software developer.

I didn’t go into this for the money – instead I wanted the freedom to work on things like Hypothesis, which I genuinely think makes the world of software development a much better place but nobody would pay me a salary to work on – but unfortunately I do still need money to live, and the financial aspect has become a significant enough source of stress that it’s come to dominate my decision making around this.

Additionally, I have the major problem that none of this is recurring – everything I’ve earned in the last year has been the result of artisanal hand-crafted contracts, largely of a form which have not been amenable to repeating. On top of that, there have been a number of potential contracts which I sunk a lot of work into getting that ended up not panning out, which is extra depressing when it happens.

The result is a weird sort of boom and bust cycle where when things are going well I’m making a tonne of money, but things are not going well often enough that it adds up to doing well in aggregate, and I can never count on any of it until the money is more or less in the bank.

Some/all of this is because I am bad at sales and marketing, and i should learn to do better at sales and marketing, but my track record so far does not suggest that I am going to do that at a rate which is sufficient to help.

The result is that this is all significantly more stressful than just earning a salary would be, and that it sucks most of my desire to do this because as well as making the whole process unenjoyable it also causes me to significantly doubt that what I’m doing is worthwhile – if people tell you that they love your stuff, but that the price they’re willing to pay for it is £0, is that really a worthwhile thing to be doing?

So it’s time to refocus. If the problem is this boom and bust cycle, the solution is to get a steady baseline. As you have probably guessed from the title, my goal for 2017 is recurring revenue.

The following are the rules that I’m setting for myself:

  • My goal is £2000/month recurring revenue.
  • My base deadline for achieving this goal is end of December 2017.
  • Every £5000 of non-recurring revenue allows me to push the deadline date back one month (but I should think long and hard about whether it’s worth doing so).
  • If things look obviously hopeless before then I may quit early.
  • If by the time the deadline arrives I have not achieved the goal, I am going to go get a normal person day job (though I won’t abandon the things I’m doing to get the recurring revenue).

This isn’t a lot of money. £2000/month is still less than my first year salary as a developer (which was £27.5k / year + a bonus scheme), but the important thing about it is that it’s more than enough to live on, which makes any revenue I make on top of that a nice bonus and makes money stop being a source of stress.

Note that the goal is recurring revenue, not passive income. I’m not looking to earn money for doing no work, I’m just looking to reliably earn money. Retainer schemes where I do, say, a day of work a month for a company sound great.

Concrete planning and how you can help

On my end, I have two or three possible projects – one in the works, one in planning, one in a vague concept stage – that should help with this. I don’t think between them they’ll make £2000/month by the end of 2017, but they might get me a decent chunk of the way there. I will promote these heavily to you when they are in a suitable state for that.

I now have a Patreon for this blog. I don’t expect it to be even close to enough to achieve this goal on its own, but in my most optimistic moments I think it might maybe get me a quarter of the way there. So if you enjoy my writing and would like me to continue it at my fairly prolific rate, please do consider becoming a patron. Note: This Patreon will self-destruct on the recurring revenue deadline date if it has not reached $500/month by then.

Finally, if you work for a software company (or any other sort of company!), do consider if there’s some sort of way I can help you on an ongoing basis – from a corporate point of view, £2000/month is peanuts. If I can get, say, five companies who are interested in paying for a day of work a month from me on an ongoing basis then I would immediately blow this goal out of the water and I could go back to worrying about more interesting things. So, think about it. Email me if you want to discuss how I can help (with no implied commitment on your part – I’m happy to just talk about it) – it doesn’t have to be Hypothesis related at all. It turns out I know quite a lot about this software lark and I’m more than happy to help people develop it in any way they’d find useful. I’d also be happy to help with mentoring new developers if you want someone to take the load off your existing senior developers.

This entry was posted in Business, life on by .

Declaring code bankruptcy for the rest of 2016

This is a small PSA.

It probably hasn’t been too visible from the outside, but I’ve not been doing very well recently.

In particular I’ve been finding my productivity has pretty much gone through the floor over the last couple months. 2016 is stressing me out for a whole pile of reasons (about 80% the same ones it’s stressing everyone else out), and I’m not dealing with it very well. This is making it very difficult to stay on track and motivated.

It’s not a problem when I’ve got something external to focus me (e.g. a client), but when I have to self-motivate on a programming project I find that I can’t right now, and the result is that I’m unproductive, which makes me depressed, which makes me even less productive.

So I’ve decided to stop. If I’m not getting any programming done and feeling bad about it, it’s clearly better to not get any programming done and not feel bad about it. Being depressed isn’t doing anyone any favours, let alone me. So, for the rest of the year I will not be writing any code unless someone is explicitly paying me to write it.

I currently have one client and a few potential clients, and my obligations to them will absolutely still be met (and probably be met a lot better than they would have in my previous mood). I am happy to accept new clients, but I probably won’t be actively seeking them out until the new year.

I’m also going to keep reviewing pull requests on Hypothesis and doing Hypothesis releases with other people’s stuff (there will probably be some Hypothesis releases with paid work from me too).

I’m probably also going to keep coding when it’s required to solve an immediate problem I have, and maybe when I need it to answer a question for a blog post or something.

So it’s not a complete cessation, but what it is is a freedom from a sense of obligation: If it doesn’t fall into one of these categories then I shouldn’t be doing it, and as a result I should be looking for something else to do if I don’t have any programming to do in one of those categories rather than procrastinating to avoid some vague sense of obligation to be coding.

This is going to give me a lot of time that I’m currently filling with failing to program in, so I’ll probably end up with some non-programming projects. I don’t yet know what these are going to be, but I’ve got a couple candidates:

  • When I first started working on Hypothesis I was taking a work break in which I’d intended to brush off some mathematics textbooks. This didn’t happen. It might happen this time. I started working through some of the exercises in Bollobas’s Combinatorics earlier and I’m finding it surprisingly enjoyable. I may keep this up.
  • I’ve been working on getting in shape the last couple of months. I don’t want to throw myself into this too vigorously because I’m more likely to just do myself an injury, but I’m likely to step this up at least moderately.
  • The War On Sleep must still be waged.
  • I’ve been thinking of doing NaNoWriMo, but I probably won’t.

Other than that, I don’t know. Watch this space while I found out.

This entry was posted in Hypothesis, life, Python on by .

The Caffeine Alarm Clock

A friend asked me on IRC for tips on waking up early, and it reminded me that there’s a thing I do that works really well and isn’t common knowledge. I can’t actually currently do it because of my current draconian restrictions on caffeine (Hi, I’m David and I am a caffeine addict. It has been 25 days since my last fix), but that’s no reason to stop you.

The trick is this: Before you go to bed, put a large glass of water and a 200mg caffeine pill  (depending on your caffeine tolerance you may wish to reduce this – it’s the equivalent of a very large coffee but isn’t an unreasonable daily intake of caffeine) next to your bed. Set your alarm for about half an hour before you want to actually get up. When your alarm goes off, take the caffeine pill and drink the glass of water.

You are now permitted to roll over and go back to sleep if you want. You are under no obligation to get out of bed.

But you almost certainly will quite soon. Within about half an hour you will be buzzing and probably need to go to the bathroom, so you won’t want to stay in bed, and you will get up at the intended time.

People look at me funny when I tell them I do this, but I think that’s just because it makes the caffeine habit a bit too transparent. People will judge you for a caffeine intake that is strictly smaller than theirs if you take it in pill form and they take it as beverages.

I am not naturally a morning person, and this won’t turn me or you into one, but as a way of getting out of bed when your first inclination on waking is that you really don’t want to, it’s incredibly effective and as long as you don’t mind the caffeine dependence I can strongly recommend it.

This entry was posted in life, The War On Sleep on by .