“Object Main extends Application” considered harmful

You’re probably all familiar with the standard Java invocation for creating an applicaiton’s start point.
class Main{ public static void main(String[] args){ // code } }

Scala has a nice feature which lets you use a trait to define your application where you don’t care about the initial arguments.

object Main extends Application{

//code

}

What happens is that Application has a method main(args : Array[String]) which gets inherited, then a corresponding static method is created on the class Main. The application logic goes in Main’s initialisation code. Neat, huh?

All we’re missing is what the definition of main is.

trait Application{

def main(args : Array[String]) = createHorrificallyConfusingBugsIfYouSoMuchAsLookAtAThread();

}

Oops.

Ok, that’s not actually true. Really it’s

trait Application{

def main(args : Array[String]) = ()

}

It doesn’t do anything. Everything happens in the object initialisation code.

Why is this an issue?

object Main extends Application{

val foo = stuff();

spawn { doSomethingWith(foo) }

doSomeLongRunningThingThatBlocks();

}

Why is this an issue?

Well, foo is a field. And Main’s constructor hasn’t completed. This means that the visibility of foo to other threads is completely undefined. Bad things will happen.

This isn’t a hypothetical scenario. We’ve just spent a non-trivial amount of time debugging a problem caused by this. We were spawning a background process from Main, and then entering QApplication.start() (the “let QT start doing stuff” method). The background process was blocking on trying to read a field from Main, and only unblocking when QApplication finally exited.

You should not expose partially constructed objects like this. It’s a bad idea, the VM will hate you for it and it will punish you with incomprehensible bugs as a result.

Edit: In fact, it turns out that this isn’t about partially constructed objects. It’s about static initialization. Because the initialization of the Main$ class depends on the construction of the Main object, Main$ will not be fully initialized until the constructor has exited. The JVM semantics guarantee that class loading is single threaded and that other threads cannot access the class until it has finished loading. Hence the observed problem.

New Blog

I’m in the process of aggregating all my old blogs together and moving them here. I realised that there a bunch of things I wanted to do that required my own site, and posts I wanted to write that didn’t really fit into my existing blog categories. Also, I really hated the name of my programming blog but didn’t feel I could change it, so this gives me an excuse to break with that.

So, here we are. I’ll be using wordpress as a general CMS for now. If I’m unhappy with it I might switch, but after some initial teething difficulties on the import it seems pretty acceptable. Certainly better than blogspot’s setup.

Expect the same old ranting about programming from me, an occasional post about food, and hopefully writing on a wider variety of topics.

Scala syntax change proposal

I came up with a neat idea for changing the syntax for call by name parameters recently (it turned out that it’s actually a reversion to an older syntax for it! The new one was there to resolve some problems, but I like the old syntax better so would rather resolve those problems directly). In the discussion of this some problems were pointed out and the feature list sortof spiralled out of control and collided head on with a previous proposal by Andrew Foggin. Here’s a summary of the current state of the proposal.

Any of the modifiers currently allowed for local variables is allowed as either a function argument or a constructor parameter. i.e val, lazy val, var or def.
A parameter marked as def has the same semantics as call by name parameters currently do. It replaces the old syntax (or, rather, the new syntax).
A function taking N arguments is equivalent to a function taking a ProductN (modulo compiler optimisations). So given def stuff(val foo, var bar, def ba z) the invocation stuff(x, y, z) is equivalent to the invocation stuff(new Product3{ val _1 = x; var _2 = y; def _3 = z; })
In order to take into account the need to make call by name parameters constructor local, and generally improve the behaviour of constructors, we introduce an additional privacy modifier, “local”. Conceptually, things marked local are only visible within the constructor. It’s basically a stronger form of private, and is the scoping modifier that constructor arguments with no qualifiers currently have. local variables and defs are not visible outside the body of the class. Unlike private members, you may not access the local variables of another member of the same class. Edit: Seth Tisue has pointed out in the comments that you can already do this. The notation for it is private[this]

Sbinary performance and Buffered IO

This is kinda a “Well, duh” moment. It’s obvious in retrospect, but I completely failed to spot it up front, so I thought I’d share.

I noticed that SBinary performance really really sucked. We’re using it at work for saving application state, and reading and writing a file of only 900kb took about two seconds! This was bad.

Some quick performance testing suggested that this was almost entirely IO bound. Reading and writing the corresponding amount of data from a byte array took fairly little time – only about 200ms for writing, 300ms for reading. So, what was I doing wrong?

After a few seconds of head scratching I realised the problem. You see, RandomAccessFile implements the DataInput and DataOutput interfaces. This is useful for small things, but for doing non-trivial binary input and output? Not so much.

The problem is that the reads and writes for these implementations are totally unbuffered. This should have been obvious, but for some reason didn’t occur to me. Oops. I’m now buffering reads and writes explicitly (currently in a pretty stupid way, but oh well). It’s a lot faster now.

SBinary backends

I’m thinking about changing the scope of some of the code for SBinary.

Specifically, you remember that part where I said “SBinary is only for serializing objects and manipulating binary data, and it’s going to remain super minimal and specialised and this will never ever change!!”? I’m thinking of changing that. :-)

The reason for this change of heart is that I’m realising how incredibly generic the constructions you put together for SBinary are. You’re basically creating a walker for deconstructing and reconstructing your entire object graph. That’s pretty damn powerful. In particular I was thinking about how to modify formats to permit sharing (another post on that will be forthcoming) and suddenly thought “haaang on a minute. I’ve written this code before”. It looks suspiciously identical to some Java code I wrote a while back for generic cloning of object graphs*. A simple rebinding of the backend to use a queue of objects rather than input and output streams would give a pretty efficient deep clone mechanism. I’ve also been thinking of creating a JCR backend which mostly works the same as the binary data (indeed, most data would probably be stored as binary blobs in the JCR), but allows for references to other nodes (and would use this for data sharing).

At the very least, this will result in ditching the explicit dependency on java.io. It will still be used extensively in the back end, but this is only visible in the API for the parts that actually need to interact with it. (most likely approach – have an Input and Output opaque type to replace DataInput and DataOutput. These will just be wrappers around the java.io types, but this won’t be visible at first)

If I do do something like this, it would still be with making binary data the priority, and there would definitely be a specialised binary frontend which should be just as convenient as the current API. If it ever looks like feature creep is threatening to destroy that I’ll separate out projects and/or cut out the idea entirely.

* In the unlikely event that anyone who worked on that project actually reads this blog, they will probably shudder in horror at the mention of that code. It was very fragile with regards to changes in the rest of the code. But that wasn’t actually an issue with the cloning – it was an issue with the post-clone processing. The graph was of database mapped objects and it needed to be partially linearised in order to insert it back into the database due to constraint issues, and this never really worked right.