# How learning Scala made me a better programmer

People will often tell you how learning functional programming / C / their current favourite language paradigm will make you a better programmer even if you don’t actually use it.

I mean, yes, I suppose this is true. It’s very hard for learning new things about programming to not teach you something general purpose, and if a language is unfamiliar enough that it’s not just a syntax change you’re almost certainly going to learn new ways of thinking along with it. I don’t know that it’s the most efficient way to become a better programmer, but it’s definitely not a bad one.

Scala, though, while there are a bunch of neat features which I still miss and occasionally emulate badly, I can point to a much more concrete and useful way in which it made me a better programmer. Unfortunately a) I don’t think it’s likely to work any more and b) You’re not going to like it.

When I was learning Scala back in… 2006 I think it might have been? Certainly early 2007 at the latest. Anyway, sure I was learning interesting things about static typing, object orientation and functional programming (maybe not much of the last one. I already knew Haskell and ML tolerably well), what I actually learned the most about was from an entirely different feature of the language.

Specifically that the compiler was ridiculously buggy.

I assume things are much better than they used to be these days, and I’m sure my memory is exaggerating how bad they were at the time, but it feels like back then the development cycle was compile, run tests, report compiler bug. I think I probably hit more compiler bugs than most people – I’m not sure why; maybe I just had a tendency to push at the edge cases. It could also be that I just was nicer about reporting them than most people, I don’t know. Either way, there was a reasonable period of time when I had the top reported number of bugs on the tracker (I think Paul Phillips was the one who ousted me. Certainly once he came along he blew my track record out of the water).

Back in the dim and distant past when I wrote “No seriously, why Scala?” someone asked the question “Why isn’t a buggy compiler a show stopper?”. There were some pretty good answers, but really the accurate one is just “Well… because it’s not?”. You pretty quickly learn to get used to a buggy compiler and integrate its bugginess into its work flow – not that you come to depend on it, but it just becomes one of those things you know you have to deal with and compensate for. It potentially slows you down, but as long as you have good habits to prevent it tripping you up and the rest of the language speeds you up enough to compensate this isn’t a major problem.

I’m not saying I’d actively seek out a buggy compiler today. Obviously if you can choose to not be slowed down this is better than being slowed down, and no matter how good your procedures for compensating for it eventually you’re going to be hit by a compiler bug in production. If I even need to say it, there are clearly downsides to writing production code with a buggy compiler.

But from the point of view of my development as a programmer it was amazing.

Why?

Well, partly just because being good at writing a decent bug report is a nice skill to have to endear you to a maintainer and this is where I learned a lot of those skills (though it took me being on the wrong end of bad bug reports to properly internalise them).

But mostly I think because just it was really useful having that much practice finding and submitting bugs. In much the same way that we spend more time reading than writing code, we also spend more time finding bugs than writing bugs. Having lots of practice at this turns out to be super helpful.

Compiler bugs have an interesting character to them. Most good advice about debugging advises you to not assume that the bug you’re experiencing is in the compiler, or even your system libraries, and that’s in large part because it indeed rarely is, so you tend not to experience this character too often.

What is that character?

It’s simple: You cannot trust the code you have written. You cannot deduce the bug by reading through the code until you get a wrong answer. The code is lying to you, because it doesn’t do what you think it does.

To an extent this is always true. Your brain does not include a perfect implementation of the language spec and the implementation of the all the libraries, so the code is always capable of doing something different than you think it does, but with compiler bugs you don’t even have the approximation to the truth you normally rely on.

“Code is data” is a bit of a truism these days, but with a compiler bug it’s actually true. Your code is not code, it’s simply the input to another program that is producing the wrong result. You are no longer debugging your code, you are manipulating your code to debug something else.

This is interesting because it forces you to think about your code in a different way. It forces you to treat it as an object to be manipulated instead of a tool you are using to manipulate. It gives you a fresh perspective on it, and one that can be helpful.

How do you debug a compiler bug? Well. Sometimes it’s obvious and you just go “Oh, hey, this bit of code is being compiled wrong”. You copy and paste it out, create a fresh test case and tinker with it a little bit and you’re done.

This will probably not happen for your first ten compiler bugs.

Fortunately there’s a mechanical procedure for doing this. For some languages there’s even literally a program to implement this mechanical procedure. I find it quite instructive to do it manually, but that’s what people always say about tedious tasks that are on the cusp of being automated. Still, I’m glad to have done it.

What is this mechanical procedure?

Simple. You create a test case for the thing that’s going wrong (this might just be “try to compile your code” or it might be “run this program that produced wrong output”. For the sake of minimalism I prefer not to use a unit testing framework here). You check out a fresh branch (you don’t have to do this in source control but there’s going to be a lot of undo/redo so you probably want to). You now start deleting code like it’s going out of fashion.

The goal is to simply delete as much code as possible while still preserving the compiler bug. You can start with crude deletion of files, then functions, etc. You’ll have to patch up the stuff that depends on it usually, but often you can just delete that too.

The cycle goes:

1. Delete
2. Run test
3. If the test still demonstrates the problem, hurrah. Go back to step 1 and delete some more.
4. If the test no longer demonstrates the problem, that’s interesting. Note this code as probably relevant, undelete it, and go delete something else.

Basically keep iterating this until you can no longer make progress.

When you stop making progress you can either go “Welp, that’s small enough” (generally I regard small enough to be a single file under a few hundred lines, ideally less) or you can try some other things. e.g.

1. Inlining imported files
2. Inlining functions
3. Manually constant folding arguments to functions (i.e. if we only ever call f(x, y, 1) in a program, remove the last argument to f and replace it with 1 in the function body)

The point being that you’re thinking in terms of operations on your code which are likely to preserve the bug. Finding those operations will help you understand the bug and guide your minimization process. Then at the end of it you will have a small program demonstrating it which is hopefully small enough that the bug is obvious.

Is this how I normally debug programs? No way. It’s a seriously heavyweight tool for most bugs. Most bugs are shallow and can be solved with about 10 minutes of reading the code until it’s obvious why it’s wrong.

Even more complicated bugs I tend not to break this out for. What I do take from this though is the lesson that you can transform your program to make the bug more obvious.

Sometimes though, when all else fails, this is the technique I pull out. I can think of maybe half a dozen times I’ve done it for something that wasn’t a compiler bug, and it’s been incredibly useful each time.

All those half dozen times were for a very specific class of bug. That specific class of bug being “ruby”. There’s a hilarious thing that can happen in ruby where namespacing isn’t really the cool thing to do and everything messes around with everyone else’s internal implementation. This potentially leads to some really bizarre spooky interaction at a distance (often involving JSON libraries. *shakes fist*). This technique proved immensely invaluable getting to the bottom of this. For example the time I discovered that if your require order was not exactly right, having a JSON column in datamapper would cause everything using JSON to break. That was fun.

But even when I’m not explicitly using these techniques, it feels like a lot of my debugging style was learned in the trenches of the Scala compiler. There’s a certain ramp up when debugging where you start with intuition and pile on increasingly methodical techniques as the bugs get more mysterious, and I think exposure to a lot of really mysterious bugs helped me learn that much earlier in my career than I otherwise would have.

It’s possible that the fact that it was a compiler was irrelevant. It feels relevant, but maybe I’d have learned variations on the technique at any point when I was daily interacting with a large, buggy piece of software. But to date, the 2007-2008 era Scala compiler is the best example I’ve had of working with such, so it’s entirely possible I’d have never learned that skill otherwise, and that would have been a shame because it’s one that’s served me well on many other projects.

# A case study in bad error messages

Consider the Python.

I’ve been writing a lot of it recently. It’s mostly quite nice, but there are some quirks I rather dislike.

You may have noticed, but I have strong opinions on error reporting.

Python doesn’t do so well on this front. This is sad given that it really loves its exceptions.

An example:

>>> float([]) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: float() argument must be a string or a number

This error message has the lovely property of being both completely unhelpful and a lie.

It is unhelpful because it does not give you any information at all (not even a type) about the value you tried to convert to a float.

It is a lie because in fact all manner of things can be converted to floats:

>>> class Foo(object): ... def __float__(self): ... return 42.0 ... >>> Foo() <__main__.Foo object at 0x27ac910> >>> float(Foo()) 42.0

I wonder how we could better design this message to mislead? I’m drawing a blank here.

# Chesterton’s patch

I encountered the notion of “Chesterton’s Fence” this morning via Ozy Frantz.

In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”

This paradox rests on the most elementary common sense. The gate or fence did not grow there. It was not set up by somnambulists who built it in their sleep. It is highly improbable that it was put there by escaped lunatics who were for some reason loose in the street. Some person had some reason for thinking it would be a good thing for somebody. And until we know what the reason was, we really cannot judge whether the reason was reasonable. It is extremely probable that we have overlooked some whole aspect of the question, if something set up by human beings like ourselves seems to be entirely meaningless and mysterious.

There are reformers who get over this difficulty by assuming that all their fathers were fools; but if that be so, we can only say that folly appears to be a hereditary disease. But the truth is that nobody has any business to destroy a social institution until he has really seen it as an historical institution. If he knows how it arose, and what purposes it was supposed to serve, he may really be able to say that they were bad purposes, or that they have since become bad purposes, or that they are purposes which are no longer served. But if he simply stares at the thing as a senseless monstrosity that has somehow sprung up in his path, it is he and not the traditionalist who is suffering from an illusion.

I would tend to agree with the quote in its original social context as a dampener on progressive change. I think most fences we’ve put up are probably silly, but I think it’s worth investigating which ones those actually are before we tear them down (and then, pass me the err. whatever tool it is you use to dismantle a fence. Lets go with “hammer”).

But reading it I recognised a very different scenario in it: What happens pretty much the vast majority of time when a programmer encounters new code. The code inevitably has special cases in it which accrued over time to handle a variety of situations. As a result the code may be quite ugly in places. The temptation is of course to rip it out, and replace it with something clean and pure and much more easily comprehensible to you, without all this cruft and special cases.

But perhaps you should first ask why is it there in the first place? And what was it intended to keep out?

# Terra: A brief review

A link that’s gone past me a few times recently is to Terra, which is “a new low-level system programming language that is designed to interoperate seamlessly with the Lua programming language”.

It looks pretty interesting. I think it perhaps make more sense to regard it as a nice library for generating C programs from Lua than as an entirely new language, but that’s both slightly splitting hairs and still pretty exciting in its own right.

Anyway, I occasionally will write low level data processing code in C, and one thing that’s always annoying there is the lack of templated container types (yes, yes, I know I could use C++. But I don’ wanna), so I thought I’ve have a go at writing one in Terra.

In accordance with my observation that I implement a hash table about every 6 months (it’s true, seriously. I don’t know why or how it happens, but if I don’t set out to do it then circumstances force me to), I decided to have a try at implementing one in Terra as a way to get a feel for the language.

Here’s the result. Be warned that it’s pretty far from a work of art – it’s the result of maybe an hour of casual hacking.

It’s a very dumb hash table, but it does the job.

What has the experience or writing it told me about Terra?

A couple things.

As a concept, it has a lot going for it. I really liked how easy it was to integrate C – you can just import C headers and, if necessary, you can inline and compile C code right on the fly (Note in the example I just embedded a C hash function. Why? Well because I couldn’t find the bitwise operators in terra and thought it would be nice to try embedding the C one as a test case). As far as FFIs go, this is a pretty damn cool one.

In terms of templating and composability? It honestly worked exactly well as promised. A++ would use again.

In terms of ease of writing? Well…

The lua bits were lua. It’s a perfectly passable language. If you haven’t used it, imagine javascript but slightly better designed. It has some quirks, but nothing worth complaining about.

The terra bits felt awfully like writing C. That’s kinda the point.

But here’s the thing. Writing C is terrifying. There’s a lot that can go wrong.

The thing that makes writing C not terrifying is that there are good tools for dealing with most of those things that can go wrong. In particular, extensive compiler warnings for static analysis and valgrind for dynamic analysis.

Terra has none of the former and appears to break the latter. As with most languages with garbage collection it’s hard to run the main lua script under valgrind, and when emitting a pure terra executable my first non-trivial terra program seems to have encountered a bug in Terra’s interaction with Valgrind (it’s possible the bug is in Valgrind rather than Terra, but that wouldn’t be my first guess).

Edit: Actually it turns out the problem is that Terra is using AVX instructions which the version of valgrind I had installed doesn’t support. Upgrading valgrind fixes this, which makes me feel much better about using Terra.

Additionally, there are a couple things that cause me to think it’s probably a lot easier to trigger these problems in Terra than it is in C. One thing that bugs me is that the mix of one based indexing (which lua uses) and zero based indexing (which terra uses) is basically an invitation to buffer overflows and off by one errors. Terra also seems very cavalier about the difference between values and pointers to values, which I’m not totally comfortable with but expect I’d get used to.

None of this is damning, but it’s enough to give me pause. It’s a lovely idea, and I hope it does well and improves, and I may well use it for some personal experiments, but right now the idea of writing anything that might be required to have any non-trivial security property in Terra would fill me with dread, which I think rather limits its use cases for me. If this isn’t something you care about, or if you have specific use cases which don’t need it, I’d definitely encourage you to check it out.

# The horror lurking at the heart of the new hypothesis

I’ve just merged a branch with a redesign of the hypothesis core into master. All the tests are passing, and I’m much happier with the new design than I was with the old one. It takes the idea of the search strategy I mentioned in my previous post and makes it the fundamental building block of the library. This seems to work quite well.

But in doing this rewrite I’ve gone and taken something which was always at the core of hypothesis and made it explicit, and it’s pretty… well actually I don’t want to say it’s bad per se. Using it has actually been very pleasant, and it’s made some very nice things possible that would otherwise have been quite awkward to do.

So while it’s not exactly bad, it’s sure as hell unpythonic.

What is it?

Well, it’s essentially a highly dynamic multiple dispatch object model with prototype based inheritance.

Err, yeah.

Why, David? Why do you do this?

Well, I arrived at it through a sequence of individually logical steps.

Here’s what drove me down this path:

The first requirement:

We need to be able to go from a type to a strategy. e.g. we want to say “give me a strategy for ints”. Or “give me a strategy for strings”. Or “give me a strategy for tuples of (int, str)”.

However, we absolutely don’t want to put the logic for this on the types themselves. We’re not going to monkey patch things to add in a strategy method or anything uncouth like that.

There are two reasons we’re not going to do that:

1. We want to be able to configure specific strategies per run, e.g. biasing for short lists or only producing alphanumeric strings
2. It would be gross

Note also that we need to be able to pass instances, not just types here. For example we can ask for a strategy for the tuple (int, str) which will give us a strategy returning pairs where the first is an int and the second a string.

So that’s where the multiple dispatch came from. It’s not implemented in any very clever way, and it currently is closer to just type based overriding than true multiple dispatch because it doesn’t support subtyping (if you define a handler for instances of Foo then it will not trigger for instances of subclasses of Foo. I may fix this at some point but I currently have no need to do so).

Where does the prototype based inheritance come in?

Well, it started at just level one: There was a single default object which all other lookups would delegate to if needed. The reason for this was that it allowed you to both define new strategy handlers wherever you wanted because you could define them on the default object, but you could then override them on a fresh local copy.

Then I realised that having full prototype based inheritance was no more work and was actually quite useful. So I provided that.

Why? Let me show you an example of its usage in practice. Here’s how we define the strategy for strings:

class StringStrategy(MappedSearchStrategy): def __init__( self, strategies, descriptor, **kwargs): SearchStrategy.__init__(self, strategies, descriptor,**kwargs) self.length_strategy = strategies.strategy(int) char_strategy = kwargs.get("char_strategy", OneCharStringStrategy)   cs = strategies.new_child_mapper() cs.define_specification_for(str, char_strategy) self.mapped_strategy = cs.strategy([str])   def pack(self, ls): return ''.join(ls)   def unpack(self, s): return list(s)

What are we doing here?

Well, MappedSearchStrategy lets you define a search strategy in terms of another one. You then just need to define methods for converting to and from instances of that other type. In this case we are defining the strategy for strings in terms of the strategy for lists of strings.

Which makes it sound like it’s recursive, but it’s not. Let me draw your attention to the following bit:

 char_strategy = kwargs.get("char_strategy", OneCharStringStrategy) cs = strategies.new_child_mapper() cs.define_specification_for(str, char_strategy) self.mapped_strategy = cs.strategy([str])

What do we have here?

We have another strategy for strings which only generates one character strings (python doesn’t have a Character data type). This is not normally installed anywhere.

We now create a new child mapper. This is a fresh SearchStrategies object whose prototype is the SearchStrategies object we started with. We can now define a strategy for strings on it, which overrides strings for that mapper but leaves the original untouched. When we ask it for a strategy for lists of strings, it uses the list strategy it’s inherited and the string strategy that’s just been defined for it to give a strategy for lists of strings that uses the new one character string strategy.

This isn’t the only place it’s used. We also use it in stateful testing in a broadly similar way for generating the individual test steps without interfering with the larger strategies namespace.

Is this all a terrible idea? I dunno. It works pretty well, and it’s allowed to write some quite interestingly powerful code, but I do have the feeling that it’s probably a bit too clever and that there might be a better solution.