My talk, Writing Libraries is Terrible, is now available online.

You can see the video on youtube, and the slides here.

Transcription

So, yeah. I’m here to tell you about writing libraries and why it’s terrible. And I’ve been basically working full time on a library for about the last year and a half, so I’ve got to experience all of this first hand.

Some caveats: I’m talking about my specific experience, so I’m talking about writing open source libraries. Closed source, proprietary, libraries, do exist. I have no experience with this at the moment, and my impression is that using those is terrible, so this is what the other end looks like.

I’m also talking specifically about my experience writing a Python library. I think very little of what I have to say is actually about Python, or Python specific, but that does colour the experiences. Different languages and different communities do respond differently.

Now, despite the title, I actually really like writing libraries. You get to focus on a very constrained problem, you get to do a lot of interesting things in the course of doing them. There’s really a lot to recommend writing libraries, and the process of writing them is enlightening and enjoyable.

But then people start using them, and that’s where things start to go a bit wrong.

This was originally going to be a talk full of technical war stories, and I’m really glad it no longer is because Jim completely topped any technical war stories I can give you, but I am going to start by just mentioning the single biggest technical problem that gives me a headache as a library author, which is that I currently support three operating systems officially and really dropping any one of these is completely non-optional [ed. I of course mean “supporting” rather than dropping], and on top of that I have to support a really large number of versions of python, and even with the ones I support people complain about the ones I don’t support. So apparently it’s really annoying that I don’t support Python 2.6 despite the fact that Python 2.6 has been end of lifed for a large number of years.

And in general, everyone’s computer is different, and as an application developer you have to deal with this and it turns out that as a library developer you have to deal with this more, because you don’t just have to worry about what’s on their computer you have to worry about the specific versions of different things. If you’re writing an application you can ship a language runtime with it and not really care which version they already have, there all sorts of advantages. I don’t get any of those.

OK, that’s all I want to say on the technical side of things, because it turns out that none of these are actually the really bad thing about writing libraries. The really bad thing about writing libraries is money.

Because, culturally we’ve just moved to this idea that of course libraries are things we get for free. Like I said, open source libraries are great to use, so of course everyone is using open source libraries. And you do get paid for open source projects, like a lot of the big ones in particularly are very well paid these days. Most people who work on Linux are paid to do so. But a lot of the open source ecosystem isn’t this sort of large open source project, it’s these single serving libraries that do one specific thing and people just install them, they don’t even really think about paying for them.

And what they pay for them instead is complaining. Like I said, people complain at me for not supporting Python 2.6, and I get some really entitled comments about it, basically just like “I’m a giant bank building legacy software in Python 2.6, making lots of money, why aren’t you supporting this thing for my for profit business for free?”, and this isn’t much fun to be on the receiving end of obviously, and it turns out to be quite hard to pay your bills with other peoples’ whining.

But actually none of this is the worst about not getting any money to work on these. Because it turns out that as well as sort of being intrinsically useful, being paid for your work gives you lots of advantages that not being paid for your advantages doesn’t have. One of the biggest, I think, under appreciated advantages of making money is that it gives you an amazingly good decision heuristic. Whenever you come to a choice, and you’ve got a choice where you’ve crossed off all the other directions – you’re ethically OK with both options, both options seem like good ideas, so on and so forth… which one of them is going to make you more money? Do I support this obscure version of Python? Well, am I going to make more money than it costs me in effort to support this version of Python? And in the course of design there are a million choices you can make in all these, and basically with open source, and with this sort of library, all that really guides your choice is do I feel like doing it and will I feel bad about not supporting this. And this turns out, if you’re like me, to be a great way of creating an amazingly large amount of work for yourself, and it’s really not obvious which of the work is important and which of the work is not important.

The other great thing about money is that you can hire people. And you do get contributions as an open source library. My project, Hypothesis, I have absolutely got contributions from people, and it’s great, but this is the amount of Hypothesis I’ve done [ed. I’m holding out my arms wide here] and this amount of Hypothesis literally everyone else has done [ed. My hands are close together now]. And I don’t hold that against them. Why should they? I’m not paying them, and basically everything they do for Hypothesis is a gift, and so I’m very grateful to those gifts, but at the same time, it’s still just me doing most of this work.

And what that means is that I have to be a lot of people. If you think about your normal software projects, you don’t just have developers, you certainly don’t just have one developer. That would be ridiculous, why would you do that? I mean, you’ve got not redundancy there, right? But you’ve also got project managers, you’ve got customer supports, you’ve got sales, you’ve got marketing, you’ve got technical writing, you’ve QA… I have to be all of those people. And… in some ways this is a good thing. I’ve really enjoyed learning a lot of these, and the best way to get a lot of appreciation for your colleagues is to try and do their jobs and discover that all the things that you thought were really easy and you thought your colleagues were kindof idiots and were just making a big deal of it… it turns out that actually people have specialised skills and some of those specialised skills are really hard. I’ve been doing a lot of attempts at marketing recently [ed. This slightly overstates the case. I’m talking mostly about hypothesis.works and some behind the scenes stuff, but I’ve been reading more] and it turns out that maybe those marketers did actually know something. This is not an opinion I would have had a year ago. But at the same time, this is really bad, because I’m not good at marketing and I’m doing the marketing for this project. I’m OK at writing, well I think I’m good at writing but I’m not good at technical writing. Documentation it turns out is an entire new skill on top of both technical understanding and writing. So I’m having to learn all of these because I don’t really have another way to do it, and I certainly can’t afford to pay anyone else to do it.

And this is all a massive problem and this isn’t going to change, because from the point of view of the people using our work, what we are basically doing is giving them money. We’re leaving this giant pile of cash on the ground and they’re just coming along and picking up because someone’s just left this giant pile of cash and obviously that means it’s free for use, right? If someone tells you that there’s this thing that is better than thing you were going to pay for and you can get it for free then of course you’re going to take it.

And this very short term thinking. As we saw in the lightning talk about CentOS yesterday, this will bite you, but by the time it’s bitten you we’ve already done all the work and it’s probably too late. So regardless of what a bad decision this may or may not be on the company’s side, this doesn’t actually help me. And this isn’t just about me obviously, this is about everyone else who also finds themselves in this position. But… in particular this is about me. And so this doesn’t help me.

And there’s really only one way this can change, which is my side and our side. At some point we’re going to have to start withdrawing our labour. If the problem is that people are taking our work and using it for free and this is really bad for us, maybe we could just… stop giving our work away for free? Is that a valid plan? It seems like a valid plan. Every time I try and explain my situation to my family they look at me like I’m crazy and ask me why I’m giving away stuff for free.

And I feel sad about this. I really like sharing my work. I’m not writing open source software because I want to make lots of money. My judgement isn’t quite that bad. I’m writing open source software because I want people to use it, and I want so share these ideas, and I really enjoy doing it.

So what I’m really hoping is that there’s some compromise we can do here. And… software licensing is a large part of this. I don’t know what part the GPL necessarily has to play in any eventual solution, but I do know that the MIT and BSD style solutions have nothing to play in this solution. They can go to hell as far as I’m concerned. And I think there’s more scope for improving the situation with licensing. And I don’t know what yet that’s going to be. I am talking to people to find out what it’s going to be. Because ultimately we do need to give people a better way to pay us, and we do need to still have ways of giving things away for free, and I don’t know where the middle ground is.

But… despite my very adversarial framing of this as us vs them, I don’t think us vs them is the solution either. Because as well as sharing, I also like making money, and I’m perfectly happy for other people to make money, and I think working together is the only way we can move forward. We have to start this, but we can’t finish this alone.

And ultimately as well as being better for us open source developers, I think it’s also going to be better for companies. Because when you think about it, 90% of the code you’re running in production has this label on it, and that’s probably a bad idea.

Thank you very much. Any questions?

[ed. I was stupid and forgot to repeat questions and we didn’t have individual microphones. I’m guessing at what people asked me based on interpretation, memory and my responses]

Question: “You must have had a plan for making money off open source software. What was it?”

Answer: I had various plans. Right now I’m mostly doing services and training. I’m going to be starting offering proprietary extensions which try and make it easy to use the core libraries for everyone, but when you start to have problems that you only really have if you’re a large company who is doing serious deployments of these things then you need to start paying me money. And there are a variety of other plans, I’m happy to talk about that at greater length afterwards. But the short answer is that it’s really hard and no-one seems to have a very successful plan yet. The best plan seems to be to do a really bad job of writing the library and then to get people to pay you to help them figure it out, and I’m not a big fan of that one either.

Question: (I have no idea what was said. I think this was mostly a question about bounty source and similar bounty programs?)

Answer: So for about the last six months a lot of the issues and feature requests on my github tracker have had the tag “for a modest fee” on them. Literally no-one has ever approached me about this. I do have a bounty source account which I need to make more obvious but I currently money from donations on that and zero money on issues. This works very well for some people but there’s this general pattern where applications with lots of consumer usage do quite well on these bounty programs more than things with companies using them. For example I know that neovim seems to be doing very well out of bounytsource right now, and I think it’s again a thing where large open source applications work much better for this than these sorts of small libraries. That is probably part of a viable future but I don’t what part it is and it hasn’t been working for me so far.

Question: Do you feel you should be paying for Python and the standard library.

Answer: Well right now I’m not a successful commercial user of it. I kinda do though, yeah. I think certainly in terms of donating to the Python software foundation that’s something I will probably do once my revenue stream starts to look slightly less pitiful and I actually start paying myself again [ed. I am now paying myself again! Or at least, I am starting doing pay roll for myself and that will at some point actually make it into my bank account], but it’s hard. There’s this huge ecosystem of open source and I don’t know yet where the money for it should come from, but I do feel that a lot more of it should come from the people who are making literally billions of dollars running off it than necessarily the individual developers who are building on top of it. But yes, it’s complex, I don’t know the answer.

Question: (Something about developers not general being buyers in a company)

Answer: Not really. I call this the “take me to your leader” problem, which is that basically the only solution I can figure out is that you need to get developers to introduce you to their managers, but that runs into the developers are not marketers problem because basically you’re trying to turn developers into your marketers and we’re not very good at that.

Question: (I have no idea)

Answer: I think there does need to be better charitable organisations but I think I’m not necessarily the person to come up with the ideas around that or to try that.

Question: Have you approached the Python Software Foundation for funding? (Something more to it than that but I’m not sure what)

Answer: The Python Software Foundation have very specific policies which is that you can apply to them for funding to work on a specific isolated chunk of work, so if there were a feature I wanted to add to Hypothesis that I wanted funding for I could potentially apply to the PSF for it, but for the general problem of ongoing Hypothesis development that’s not the sort of thing that they fund, and so I’ve largely not looked into it past that point. But for certain things that’s absolutely a thing you should think about doing. And also there are within specific areas there are other funding bodies, so for example for Django there’s the Django foundation who can do this, there’s NumFOCUS who tend to focus on scientific Python, and a bunch of relatetd things. So sometimes that works, but it tends to be more solutions to specific problem than the general larger problem.

Question: (I think about the idea of my writing a book)

Answer: Yes I’m going to look into that, because I’ve already a huge number of articles and so I figure I can turn those articles into a book. But that’s more marketing than it is going to be necessarily a useful income stream. I’m expecting that to make some money, and this is the sort of solution that only works once you’ve put in all this work and done all this free work, so again I think writing books is… it’s a great partial solution to this problem but it doesn’t actually solve the problem. And maybe with all the partial solutions available if you do literally all of them it starts to add up to a whole solution, but right now I’m not convinced it does.

Question: Is open source possible the wrong answer to this? Or is just a marketing strategy?

Answer: I don’t know if open source is the wrong answer. I think that closed source is definitely the wrong answer, and I think that finding some hybrid between the two, but as we saw in Jim’s talk earlier and in plenty of others, there’s a reason why open source is so popular and it’s not just that stuff is being given away for free. Your life tends to get very bad if you build your things on a foundation of other peoples’ closed source products. And this sort of free sharing of ideas really does have a lot of benefits, so I don’t want to throw the baby out with the bathwater. I really like open source and want it to succeed, and that’s why I’m trying to find a compromise. In terms of the question of whether it’s marketing or not, marketing is definitely a component but I think we need more than just marketing. I think no matter how well you market your open source project that’s not going to make you any money unless you also have a path to making money from it. Which is what I’m trying to figure out.

Question: Something about using the open source product as the marketing for something.

Answer: That is the open source core, or something like that, and I think that’s a viable solution, but you do then end up with… in that case you’re sort of asking “how much of the benefits of open source do you want to get?” and I would like something where you could get both the benefits of open source and the benefits of commercial work. I would also like a pony.