Category Archives: Uncategorized

Getting Google Analytics on readthedocs

If you’re like me, you like the following things:

Knowing about people who use your software
Not hosting your own documentation

As a result, you want some sort of analytics on readthedocs!

If you’re very like me this will have resulted in you spending a fruitless morning looking through the results of google and finding none of the solutions work at all.

Well, allow me to fast forward this process for you. The correct way to add google analytics to your readthedocs instance is as follows:

Go to the Admin page for your project
Click on “Advanced settings”
Enter your Google Analytics Tracking ID in the box at the bottom that says “Analytics ID”

That’s it.

I have literally no idea how you are supposed to discover this. I found it by accident while messing around with build settings to try to get one of the other solutions to build. Hopefully this is of use to you.

How to improve your Quickcheck implementation with this one weird trick

When asked what’s genuinely novel in Hypothesis as opposed to mere optimisation, I usually cite three things:

Strategies carry simplification and serialization with them but still compose monadically.
The data generation can hit corners of the search space that standard Quickcheck can not, and adapt dynamically to requirements.
The simplification is much more sophisticated than the standard Quickcheck interface and can adapt to the structure of the problem much better.

It took me a surprisingly long time to notice that all three of these rely on essentially the same tweak to the standard API.

The tweak is this: Introduce an intermediate representation.

By introducing an intermediate template type which is then reified to the final result, you can simplify a mapped object by simplifying the argument.
Instead of generating a random value, you first generate a random parameter and then draw a conditional value given that parameter. As well as producing more interesting results this way (because of how you compose parameters for e.g. lists) this also lets you reuse parameters which are more likely to give you a useful result.
Instead of having a single simplify function, you have a list of simplifiers which you apply in turn.

Each of these have taken what was a single step and broken it up into two steps which you can naturally compose to give the original. We then don’t compose these in the obvious way, but instead change things around just enough to give something strictly more powerful while still mostly retaining the original characteristics.

I’m not sure what the broader significance of this is yet, but it seems likely that this approach has other areas to which you could apply it.

Some empirically derived testing principles

Here are some principles I have found useful when testing Hypothesis. I don’t promise any of these are universally good, all I promise is that all of them have resulted in my finding bugs that I would otherwise have missed, and that together they seem to give me a lot more confidence in my software than I otherwise would have.

1. 100% coverage is mandatory.

This is a slight lie in that I do have the occasionally nocover or ignore pragma sprinkled around the code, but those generally occur in places that I’ve tried really hard to cover and it’s just not possible to hit reliably.
People seem to think that coverage is a bad metric and doesn’t really tell you a lot about your testing. There’s a line I have here which I believe is stolen from someone from the sqlite project:

100% coverage tells you nothing, but less than 100% coverage tells you something.

Code that is not tested almost certainly contains bugs. Code that is tested probably still contains bugs, but the probability is lower. Therefore, all other things being equal, code with 100% coverage is likely to contain fewer bugs.
This is essentially operating on the principle that metrics make great constraints and lousy goals. The goal is not to maximize coverage, the goal is to test our project well. 100% coverage is the starting point, not the finish line.
(“Coverage” here means “branch coverage”. In reality it means “The finest grained coverage metric that you can get out of your tooling and that it’s meaningful to claim 100% coverage on”)

2. All bugs result in tests.

The core principle of this is hopefully both self-explanatory and uncontroversial. Step 1 to fixing a bug is to write a test for it. Otherwise the bug might reoccur.
But it goes deeper. Every bug is actually two bugs: A bug in your software, and a bug in your test suite.
The former is obvious. A bug in your software is a bug in your software. The latter is more subtle: It’s a bug in your test suite because your test suite didn’t find this bug, and that indicates a failure of your testing.
So. You’ve found a bug. Why didn’t your test suite catch it? What general failure of testing this indicates, and can you fix that failure in a way that would catch other instances of similar bugs?
As a concrete example: In Hypothesis’s data serialization layer, one of the invariants is that an attempt to deserialize bad data can only raise a BadData exception. Any other exception is a bug. This invariant is used to guarantee that Hypothesis can always read from old databases – the worst case scenario is that you can’t reuse that data, not that it crashes your tests.
In preparing for the 1.1 release I found an instance where this wasn’t the case. This then caused me to write some generic tests that tried fuzzing the data Hypothesis reads in order to find exceptions that weren’t BadData. I found 5 more.

3. Builds should fail fast.

There’s nothing more frustrating than getting to the end of a long build and then having the trivial thing that you could have found out 30 seconds into the build failing everything.
Linting is a major culprit here. lint should run at the beginning of builds, not at the end. I also have a separate entry in the build matrix which runs only a fast subset of the tests and checks the results for 100% coverage. This means that if I’ve forgot to test some area of the code I find out fast rather than at the end.

4. Principles should never be satisfied by accident.

Suppose your test suite catches a bug in your change with a failing test. Is that test actually the one that should have caught it? This particularly comes up with internal APIs I find. Testing internals is important, and if a bug in an internal was only caught because of a bug it caused in the public API, that internal could use a more specific test.
This also plays well with the faster build step for coverage. By forcing coverage to happen on simple examples it ensures that code is covered deliberately rather than as a result of testing other things. This helps offset the problem that you can often get to 100% coverage without ever really testing anything.
Flaky tests are often an example of something happening by accident. Sometimes it’s just that the test is bad, but often it’s that there’s some hard to trigger condition that could use a dedicated test that hits it reliably.

5. You need both very general and very specific tests.

Tests need both breadth and depth. Broad tests are things like Hypothesis which ensure that your code is well behaved in every scenario (or at least as close to every scenario as they can), but they tend to have very shallow definitions of “well behaved”. Highly specific tests can assert a lot more about what the code should do and as a result are true in a much narrower set of environments.
These tend to catch different classes of bugs.
So far to the best of my knowledge nobody has come up with a way of testing things that achieves both breadth and depth at the same time other than “do a massive amount of work manually writing tests”, but you can do both by writing both kinds of tests, and this is worth doing.

6. Keep trying new ways of testing things.

It’s been my experience that every time I try to come up with a new way of testing Hypothesis I find at least one new bug in it. Keeping on coming up with new and creative ways of testing things is a great way to keep ahead of your users in finding bugs.

7. There is no substitute for hard work.

Unfortunately, quality software is hard. I spend at least as much effort testing Hypothesis as I do writing it, and there’s probably no way around that. If you’re taking quality seriously, you need to be spending as much work on quality as functionality.

Notes on pilling a cat

This little monster is called Dexter.

He’s a lovely cat, really. A bit whiny, slightly skittish, but otherwise very nice. He likes skritches and laps and is generally very polite when there isn’t a door he wants to be on the other side of.

Unfortunately he also has a variety of exciting bladder problems. He’s currently on a five pill a day regime in an attempt to sort this out.

Dexter does not like pills.

We are lucky in that one of the things Dexter does not do is attack people. He might injure you in an attempt to get away when sufficiently panicked (ask me about bath time for kitties), but he never actually attempts to harm you no matter how much he disapproves of your actions.

Nevertheless, we’re engaged in an arms race. It used to be the case that I could just cradle him in my arms, stick a pill in his mouth, and he would swallow.

That is, uh, not the case any more.

Here is our current cat pilling protocol, presented in case it will be useful to others.

You will need:

Two people. Don’t even think about trying this with one. We are considering whether adding a third person to the protocol would be useful.
All of the pills extremely conveniently to hand.
A dropper pipette with water in it. A syringe would also work but you need to be good at not squirting it excessively.

There are two people in this protocol. For sake of avoiding ambiguity, I will refer to them as David and Dave. David is sitting, Dave is standing. The role of David is cat control, the role of Dave is pill application.

David cradles the cat in his arms lovingly. He pretends this is to lure the cat into a false sense of security, but actually the cat knows exactly what is about to happen.
David sits down with the cat flat on its back and head pointing away from him, wisely exposing his belly to the sharp back legs (you may wish to take protective measures if you have a less polite cat than Dexter).
David grabs the cat by the scruff of the neck. If the cat does not look in mortal terror for his life at this point from how much his face is being pulled back, David has not grabbed enough scruff.
David traps the front paws, which the cat has learned to move while scruffed, under an arm and levers the cat’s jaws open.
Dave swiftly moves in, drops a pill into the back of the cat’s throat, and follows with a squirt of water.
David holds on very tightly. The first time we squirted water into Dexter’s mouth he somehow teleported across the room.
David prises open the cat’s mouth to check that the pill has indeed been swallowed.

Something about the squirt of water seems to cause Dexter to instantly swallow. This appears to be a general phenomenon according to the internet.

In order to understand these drastic measures, here is a list of things that don’t work any more or never worked:

Dexter is terrible at eating. He doesn’t really do it. He certainly doesn’t do it if you try to feed him from your hand. As a result none of the standard “wrap the pill in food” measures have ever been effective on him.
Cat burrito is not a useful control procedure for Dexter. We used to try to use this to control him, but it just freaks him out more because he knows exactly what’s about to happen. We end up having to reset the burrito between each pill, which causes him to freak out more. It’s easier to just hold his legs in place with body weight and, if necessary, hands.
Releasing his scruff, holding his mouth shut and stroking his throat results in him very helpfully makes swallowing motions. You then later discover he’s stored the pill in his cheek.
Anything that allows him to move his jaw. He chews up the pill and starts foaming pill residue at the mouth. If we didn’t know the cause of it, the sight of our cat foaming red at the mouth would have been quite unsettling.

I expect the current protocol is only going to work for so long before Dexter figures out how to not swallow when you squirt water into his mouth. Fortunately I expect to have moved out by then.

Edit to add: I had to pill the cat solo tonight. The burrito technique worked… well at least it worked better than trying to handle the cat without the towel. The experience was significantly less pleasant for both human and feline participants, but I didn’t have enough hands to do it any other way.

Important things I have learned about Twitter on Android today

You know that thing where the Android twitter account wants to “use your installed apps to customize your experience”? The latest in privacy intrusion features from the app that also wants to upload all your contacts to Twitter’s servers and will require you to remember to uncheck the box literally every time you set up a new account on any device ever if you don’t want your entire address book to be shared with them.

Well, as with all of the other obviously global settings on Twitter’s Android app (such as “No you may not beep at me. Oh god why are you beeping. Stop, please.”) it’s a per account setting. If you have multiple twitter accounts set up on the app, you need to turn off the “It’s OK to spy on the rest of my phone” on each one.

This was finally enough to prompt my growing desire to uninstall this app, so I did.

At which point I was reminded that Twitter’s mobile website is of the “Hey? Did you know we have an app? Have you forgotten since the last page that we have an app? Maybe you’d like to install our app? Have you maybe changed your mind about maybe installing our app? How about now?” school of design.

Which is why the second important thing I learned is that if you want to not be constantly annoyed by the only way to use Twitter on your phone without it trying to take all the rest of your data along with it, the solution is to click the “Request Desktop Site” checkbox in the settings menu. You’re on the mobile subdomain, so you’ll still get the mobile site, but the UA won’t say you’re on Android so you won’t get the constant needy requests to install the app.