As you might have noticed, testmachine development stalled for a bit there. It wasn’t through a lack of interest – it’s been a mix of illness, busyness and being stuck on a thorny problem.
The thorny problem in question being subprocesses. It’s really important to me to be able to run testmachine programs in a subprocess, because my major use case for it right now is testing intmap, which is full of internal C assertions. One of the major benefits of generating tests with testmachine is that it can find ways to trigger those assertions (fuzzers love assertions), but if the process crashes before they can usefully reduce the test case the results are… unhelpful.
The reason this has been a hard problem is that testmachine is quite stateful, and synchronizing that state across the process boundary is quite painful. I knew how to do it, but it was going to be a lot of work and felt very inelegant.
Fortunately, on Wednesday, I realised I didn’t have to!
Despite the relatively constrained main interface to it, the internals for testmachine are pretty general – there’s a context object which gets passed around and maintains a bunch of state (a number of stacks of variables, and a random number generator), and languages and operations are essentially free to manipulate that state arbitrarily.
Well, I’ve added two constraints to that which have made life vastly easier:
- A language may only depend on the heights of each stack, not the values in them
- An operation must be able to predict its stack effect without knowing what the values on the stack are (by providing a simulate method)
Given these two constraints the result is that we can simulate the execution of a program in a safe way without invoking any of the dangerous functions – we can run a version of it that calls simulate instead of execute, accurately maps the variable reads and assignments of the program, and provides us with a nice pretty output without having to worry about our program randomly crashing in the midst of its compilation process. We can then run the version that actually executes the code in a subprocess and safely let it crash.
I would expect the new feature to be a little ropey right now. I’m aware of at least one problem with it that I’ve yet to fully resolve (It makes assumptions about how you use the randomness that hold if you use the main interface but are not guaranteed by the internals). But it passes testing, is off by default and is a pretty useful feature, so I’m pushing it out now. Let me know if you have any problems.