It’s commonly claimed that every language has a Quickcheck port. This is one of those statements that is simultaneously true and really unhelpful. Every language has a Quickcheck port that you really don’t want to use. Some languages have Quickcheck ports that you do.
I thought I’d do a quick survey of the current state of play for different languages.
First, a disclaimer: I do not write all of these languages, and many of them I barely read. As such, some of this list is based on an outsider making conjectures and if you do write the language and have better information than me I would very much appreciate correction.
First, lets start with the “just use this one” table. These are languages for which there is more or less unambiguously a single Quickcheck you should be using.
Minimum criteria for being on this list:
- Must support random generation of data to a test function. Note that I consider Smallcheck its own thing, so Smallcheck ports don’t count for inclusion unless they also support random generation.
- It must be fairly straightforward to generate your own custom types
- It must support shrinking
- It must either be under active development or have a reasonable claim to being finished.
- Must be under an OSI approved license (note: I haven’t checked this one too carefully, but these all claim to be open source and I’ve checked the license in detail on about half of them).
- There isn’t another implementation for the language also satisfying the above 5 conditions which also gets a comparable amount of active use.
Note that the presence of an item on this list does not mean that I have used it, only that I have inspected it to see what it supports. In particular I can’t comment on the data quality or how buggy or stable each of these is, though most of them appear to be pretty good.
|C||theft||Provides nothing in the way of built in generators. You’ve always got to write your own from scratch. However the API for doing so is pretty easy.|
|Haskell||Quickcheck||The original. Probably no longer the best.|
|Python||Hypothesis||You knew I was going to say this, so don’t even pretend to be surprised.|
|Ruby||Rantly||I really wanted to disqualify this one on grounds of the way that you support shrinking is by monkey patching a ‘shrink’ method on to the type. I couldn’t quite justify doing so because monkey patching common and easily clashing words onto types is the Ruby standard idiom. I am unclear on how active development on Rantly is.|
You may have been surprised that Erlang is not in the table despite being one of the most well known languages for Quickcheck. The reason is that there isn’t an unambiguous choice there. There is eqc, which is unambiguously the best but is not open source, and there’s Triq and PropEr, both of which are under active development. Apparently people use PropEr unless they have to avoid the GPL, in which case they use Triq.
As far as I can tell, people who want good Quickcheck for Java write their tests in Clojure or Scala. There is junit-quickcheck which is disqualified on grounds of not implementing shrinking. functionaljava does have a Quickcheck implementation that supports shrinking, but it seems unclear that people actually know this exists and are using it. It is possible it should be on the above table, but I’m softly considering it to fail condition 6 because test.check and ScalaCheck just seem to completely dominate it. I am very open to arguments that this is not the case.
Similarly, other .NET languages just seem to adopt the policy that you should just use FsCheck to test your code rather than trying to port it to the current language.
Go has a deeply inadequate version of Quickcheck which doesn’t support shrinking baked into its standard library. This is quite surprising as Quickcheck is from the 90s rather than the 70s, but they make up for it by not acknowledging where the concept is actually from. There do not appear to be any alternatives.
OCaml seems to be suffering from a problem of being close enough to Haskell that people try to do a straight port of Quickcheck but far enough from Haskell that this doesn’t work. The result is that there is a “mechanical port” of Quickcheck which is completely abandoned and a fork of it that uses more idiomatic OCaml. I am insufficiently familiar with OCaml or its community to know if either is used or whether there is another one that is.
There are of course many other languages than this, any many of them have something that looks a bit like Quickcheck if you squint hard enough. The Quickcheck page on Wikipedia lists a bunch of others, but a not entirely thorough sampling of the remainder suggests most of them fail the test of “does it implement shrinking” and the test of “is it actively developed or finished” and are not widely used enough to be worth an honourable mention.
If you know of a reasonably good Quickcheck that I have not included here, please do let me know.