Planet Scala

Manifests: Reified Types

2008-12-03T08:37:00+00:00

Reification (n.): When an abstract concept [...] is treated as a concrete 'thing'.^[1]

When Java-the-language added generic types in version 1.5, Java-the-virtual-machine (JVM) did not. Generic types are a fiction of the compiler. They exist at compile time, but they are omitted from the generated bytecode and are therefore unavailable at run time. This phenomenon is known as erasure.

By contrast, when C# added generic types in version 2.0, the .NET Common Language Runtime (CLR) added them as well. This allows C# to access information about generic types at run time. We say that C#'s generic types are reified.

Java's type erasure leads to some peculiar limitations. To deal with these limitations, Java programmers often find themselves passing around java.lang.Class<T> objects: concrete manifestations of the otherwise abstract concept that is the type "T".

Starting with version 2.7.2, Scala has added manifests, an undocumented (and still experimental) feature for reifying types. They take advantage of a pre-existing Scala feature: implicit parameters.

To use Manifests, simply add an implicit scala.reflect.Manifest[T] parameter to your method, like so:

def name[T](implicit m: scala.reflect.Manifest[T]) = m.toString

This method prints the name of any Scala type. To use it, invoke the method with some type parameter:

name[Int => Int]
// returns "scala.Function1[int, int]"

When using implicit parameters, you usually have to declare a implicit identifier with the same type as your method expects, in the same scope where the method is called. With Manifests, the compiler automatically injects the implicit parameter for you, as long as it has enough type information to generate a Manifest. Essentially, this is a way of carrying over type information available at compile time into objects available at run time.

An Example: Generic Collections Optimized for Primitives

In Java, generic collections can't be used with primitive types. The boxed types must be used instead. This has implications for both run time performance and memory consumption. Despite HotSpot's best optimization efforts, boxed types can add significant overhead in performance-critical code.

In C#, by contrast, generic collections are optimized for performance with primitive types. This is possible because C#'s types are reified. If your code calls for a List<T>, C# knows at run time what T is and can use the specific implementation of List that is optimized for your T.

Let's see what Scala can do with Manifests and a lot of help from the fastutil library, which provides 1000+ (!) Java collection classes optimized for primitives.

import scala.reflect.Manifest
import it.unimi.dsi.fastutil._

def mkList[T](implicit m: Manifest[T]) = {
  (m.toString match {
    case "boolean" => new booleans.BooleanArrayList
    case "byte" => new bytes.ByteArrayList
    case "char" => new chars.CharArrayList
    case "double" => new doubles.DoubleArrayList
    case "float" => new floats.FloatArrayList
    case "int" => new ints.IntArrayList
    case "long" => new longs.LongArrayList
    case "short" => new shorts.ShortArrayList
    case _ => new objects.ObjectArrayList[T]
  }).asInstanceOf[java.util.List[T]]
}

Now, invoking the mkList method with the proper type (i.e., mkList[Int], mkList[Double], etc.) will create an ArrayList optimized for that particular primitive type. We can verify that the correct classes are being instantiated by doing a little reflection.

mkList[Int].getClass.getName
// returns "it.unimi.dsi.fastutil.ints.IntArrayList"

mkList[Double].getClass.getName
// returns "it.unimi.dsi.fastutil.doubles.DoubleArrayList"

These optimized collections can have considerable benefits over their unoptimized counterparts in java.util. Just imagine, for example, the memory overhead of storing boxed Bytes instead of unboxed bytes. Storing just the pointer for one boxed Byte takes up at least four (and possibly eight) bytes!

Of course, the same effect could be achieved in Java by passing an explicit java.lang.Class<T> parameter to mkList. The disadvantage is that this imposes a significant overhead on the programmer to create and pass around these Class<T> objects. Scala's support for implicit parameters, and the automatic injection of Manifests by the compiler, makes reifying types on the JVM slightly less painful.

Appendix: Other Methods on Manifests

In this example, the only method we used on Manifests was toString. What other methods are available? Manifests are as-yet undocumented (not in the official Scaladocs), but looking through the source can give us an idea of what's available.

erasure

This method returns the java.lang.Class that corresponds to the run time erasure of the Scala type represented by the Manifest.

<:< and >:>

These methods test whether the type represented by the Manifest is a subtype or a supertype (respectively) of the type represented by the given parameter. As the source warns, the current implementation is merely an approximation, as it is based on the erasure of the two types.

equals

This method tests whether two types are equal. Again, the source warns that the current implementation is merely an approximation, as it is based on the erasure of the two types.

Thanks to David Hall for suggesting the example.

scala.Function1 lacking

2008-12-03T07:36:51+00:00

The Scala API leaves a lot to be desired. I’m going to pick on a few methods that should appear, but do not, on scala.Function1.

They are:

map
flatMap
*> (the S combinator)
on

Using some magic with the implicit keyword I can make it appear as if these methods did in fact exist:

sealed trait RichFunction1[-T, +R] {
  def apply(t: T): R
 
  import RichFunction1.rich
 
  def map[X](g: R => X) = rich[T, X](g compose (apply(_)))
 
  def flatMap[TT <: T, X](g: R => RichFunction1[TT, X]) =
    rich[TT, X](t => g(apply(t))(t))
 
  // The S combinator (SKI)
  def <*>[TT <: T, X](f: RichFunction1[TT, R => X]) = (t: TT) => f(t)(apply(t))
 
  // S again, swapped arguments
  def <*>:[TT <: T, X](f: RichFunction1[TT, R => X]) = <*>(f)
 
  // map with swapped arguments
  def <-:[X](g: R => X) = map(g)
 
  def on[K](f: (R, R) => K, t1: T, t2: T): K = f(apply(t1), apply(t2))
}
 
object RichFunction1 {
  implicit def rich[T, R](f: T => R) = new RichFunction1[T, R] {
    def apply(t: T) = f(t)
  }
}

By having flatMap (and therefore map) this allows you to remove a lot of duplication. This may come at the expense of syntactical noise per Scala, but not always. Suppose you were given a String and you wanted to check if it was equal to one of a few Strings (ignoring case). You could use some trickery with existing methods on List, but I want to keep this example simple, so let us ignore that possibility for now (and accept that I could come up with a sufficient example that such trickery is insufficient).

  // For example, suppose this predicate function
  def predicate(s1: String) = s1 equalsIgnoreCase (_: String)

Here is the repetition

  // predicate(s, _) repeats
  def f(s: String) = predicate("x")(s) || predicate("y")(s) || predicate("z")(s)

But if we have flatMap and map we can use a for-comprehension:

  // Taking advantage of flatMap/map
  val g = for(a <- predicate("x");
              b <- predicate("y");
              c <- predicate("z"))
          yield a || b || c

Here is how that same code looks when expanded:

  // Expansion of g
  val h = predicate("x") flatMap (a =>
          predicate("y") flatMap (b =>
          predicate("z") map ((c =>
            a || b || c))))

How about some fancy stuff with the S combinator (<*>):

  val or = Function.curried((_: Boolean) || (_: Boolean) || (_: Boolean))
 
  // Using the S combinator
  val i = predicate("z") <*>
          (predicate("y") <*>
          (predicate("x") map
          or))

Or with the arguments swapped around:

  // Using S with swapped arguments
  val j = ((or <-: predicate("x")) <*>: predicate("y")) <*>: predicate("z")

Pretty neat eh?

Suppose you wanted to check if the length of one List was less than the length of another. You might be tempted to write x.length < y.length. Notice how _.length repeats? Again I want to keep this example simple so while the solution below is more noisy, there are cases where it is not.

Scala is let down a little by first-class function semantics. We’ll begin with assuming this first-class function value:

  val length = (_: List[Int]).length

Then comparing using length:

  val k = length on (_ < _, List(4, 5, 6, 7), List(1, 2, 3))

A bit noisier but the repetition is gone. It’s a shame that abstraction comes at a syntactic cost and in some cases it may even be worth that cost. I wish I had the choice.

G1 Camera Tricks

2008-12-02T23:25:08+00:00

The G1's camera seems to be slow. Really slow. As in several-seconds slow. I always wondered why; then I started to look at my pictures from the Macy's Thanksgiving Day Parade.

Ah. I see. I don't think that's because of the curvature of the lens (it's not curved THAT much) I was apparently panning the camera when I took this picture (to keep up with the float, though I didn't correctly account for the very long shutter delay, and the float ended up behind the pole). You are seeing the effect of the CCD "scanning". This reminds me of this project, in which a dude builds a digital camera from a flatbed scanner. This is a picture of his garage door opening and closing while he takes a single exposure.

I hope this isn't an actual limitation of the G1's hardware. My cheap Nokia 6267, a feature phone, had an excellent 2MP camera that took pictures nearly instantaneously. So far, the G1's camera reminds me of my short experience with Windows Mobile 5's camera app - slow and clunky.

Using a Horizontal Progress Bar in Android

2008-12-02T23:05:13+00:00

I was working on my Android downloader application yesterday, when I ran into a bit of a snag. I wanted to display a list of downloads (similar to the Firefox downloads window). Each item in the list should show the file name and a horizontal progress bar. I started to implement this, and several hours later stumbled upon the archaic solution. I wanted to share this, both to help anybody who may be trying to do something similar, and also to illustrate how the lack of good documentation is hurting aspiring Android developers.

I thought that I could put a ProgressBar instance in my xml, set it to be indeterminate, give it a max and a value, and be done. However, it wasn't nearly that easy. After a few hours of hunting on Google and the Android Groups, I stumbled upon the solution in an online preview of The Busy Coder's Guide to Android Development. In short, I had to set the following on my ProgressBar's XML:

style="?android:attr/progressBarStyleHorizontal"

WTF?

Let's try to break this down. The ? on the front indicates that this is a reference to something in the current theme. What is a theme? Well, Android's UI infrastructure is similar to Swing or HTML in that it tries to accommodate components whose size is not known until runtime. Android's components are also themeable (similar to the Swing Synth LAF). A theme is a collection of related component styles. You can control the theme used by your application in the AndroidManifest.xml file. So, I think we can safely conclude that attr/progressBarStyleHorizontal is something whose actual value is defined in a particular theme.

If you take a peek in the android source code, inside frameworks/base/core/res/res/values/attrs.xml, you will find the following definition:

<resources>
    <declare-styleable name="Theme">
        <!-- snip -->
        <attr name="progressBarStyleHorizontal" format="reference" />

OK, not very useful. However, in frameworks/base/core/res/res/values/themes.xml, we find:

<resources>
    <style name="Theme">
        <item name="progressBarStyleHorizontal">@android:style/Widget.ProgressBar.Horizontal</item>

Nowe we're getting somewhere. Finally, inside frameworks/base/core/res/res/values/styles.xml, we see:

<resources>
    <style name="Widget.ProgressBar.Horizontal">
        <item name="android:indeterminateOnly">false</item>
        <item name="android:progressDrawable">@android:drawable/progress_horizontal</item>
        <item name="android:indeterminateDrawable">@android:drawable/progress_indeterminate_horizontal</item>
        <item name="android:minHeight">20dip</item>
        <item name="android:maxHeight">20dip</item>
    </style>

What's going on here? Well, it looks like themes.xml defines a theme as a collection of styles. Along with that, styles.xml specifies a set of attributes that should be applied to a particular XML element. By saying

<ProgressBar style="?android:attr/progressBarStyleHorizontal">

we are really saying

<ProgressBar style="@android:style/Widget.ProgressBar.Horizontal">

which is like saying

<ProgressBar 
    android:indeterminateOnly="false" 
    android:progressDrawable="@android:drawable/progress_horizontal" 
    android:indeterminateDrawable="@android:drawable/progress_indeterminate_horizontal" 
    android:minHeight="20dip" 
    android:maxHeight="20dip">

Spiffy.

For bonus points, we can look in frameworks/base/core/res/res/drawable/progress_horizontal.xml. This file apparently contains the declarative expression of the horizontal scrollbar graphic. I don't know how extensive this syntax is, but a quick peek at progress_indeterminate_horizontal.xml indicates that it is possible to specify animations in this XML file format.

All this seems really powerful, only somewhat useful, and totally confusing. All I wanted was to create a horizontal progress bar. I can't tell why the Android widget collection includes only one ProgressBar class that is meant to be used for horizontal, indeterminate horizontal, indeterminate circular, and every other kind of possible progress indicator. Perhaps there would be duplication of code if those were separated into different classes. Whatever. I actually don't mind that they stuffed all those progress indicators into a single class. There are some questionable decisions (like having both indeterminate and indeterminateOnly), but I can deal with it. On the other hand, the incantations that you have to use to get the Progress Bar to behave are truly archaic. I think I have proven in this blog post that there is some method to the madness and, as is often the case, easy access to the source code is a must. What bothers me most is that there was absolutely no documentation on the subject. Google seems silent on this topic, and even The Busy Coder's Guide mentions that the style attribute won't be covered until a later edition of the book (though it is possible that the Google Books sample chapter is out of date at this point).

I still maintain that Android is one of the most unique application systems that I've seen. However, as I've tried to develop for it, I have encountered a general lack of polish. Many things that you want to do as a developer are confusing and poorly documented. Familiarity with existing technologies (such as Swing or Applet programming) is pretty much useless here. On the other hand, the core concepts (such as the use of Intents to start applications and communicate between processes, the HEAVY use of message passing, and the declarative nature of the AndroidManifest.xml file) seem to be an excellent base upon which to build not just an open operating system, but a truly open user experience. A power user can really replace virtually any aspect of the system. Perhaps developers will create third party libraries that make Android a little easier within which to develop, or maybe Google itself will provide simpler interfaces for developers. Or perhaps Android will die out because developers had a difficult time building anything particularly complex. Whatever the outcome, I'm excited for the ride.

Debugging inside the Scala Eclipse Interpreter

2008-12-02T17:32:36+00:00

So today we didn't have a babysitter, so I took the day to spend time with my daughter. It was quite fun, and I actually had a chance to get to some of the scala development work I've been meaning to during her nap.

Specifically, there have been a bunch of bugs with the Scala Eclipse plugin that really fall under contributions I've submitted. Primarily relating to the Interpreter.

After spending quite a long time getting my SDT development workspace working again (seriously... I have no idea why it's so hard to make it work on windows, but it is), I was up and running. In about 20 minutes I managed to convert the original "roll-my-own jvm executor" to use Eclipse JDT's "Launch Configuration" support for a JVM. Not only does this let people specify nice things like JVM arguments, it also rolls in the ability to actively debug.

Here's a screenshot of a workspace working off my locally installed scala plugin:

Hopefully this will make it into the trunk in a few weeks, depending on how much free time I can find. There's still a few things I need to work out such as:

Wiring the old menu-style invocation to work with the new LaunchConfiguration

Fixing/Tweaking the Launch Configuration Dialog to get rid of wonky behavior and allow configuration of things that matter

Allow "bootstrap" code to send to the interpreter

Ensure jline works correctly and eclipse passes down "interesting" keystrokes, OR roll my own

Figure out how to wire into the syntax highlighter for a console

As you can see... it's already an extensive list. I figure I'll at least get the basics done for the next release (2.8 I guess?)

Anyway, if anyone wants to try out the code locally, let me know and I'll send you a .patch.

Cross-platform dressing not always a travesty

2008-12-02T14:07:00+00:00

Friday is a news reader that is fed by an obscure, non-Google service known as “NewsGator,” as explained in last week’s post. And so many things have happened since then! The technology that drives Friday’s visuals, Processing, had a splashy 1.0 release. The Thanksgiving holiday was observed in the United States and we proved once more that our domestic travel infrastructure does not scale. Then on “Black” Friday, a Wal-Mart employee was trampled to death in a stampede of discount shoppers. All the while our Friday related these stories with unflappable grace and charm.

In celebration and exhaustion, let’s take this opportunity to write an easy post with lots of pictures.

Platform swingers

In the opera of Java, the tragicomic figure of Swing is rivaled in pathos only by El Duque himself (riding a swing). Swing was conceived in the midst of an object-oriented resource-based interface revolution, but the toolkit somehow managed to escape the paradigm that would have made it an unmitigated success. Instead, it opted for interface definition in code, which is great for writing Hello World and terrible for Usable Application. Ten years later Swing is popular for internal corporate apps that employees must use, or be fired.

Processing doesn’t advertise any support for traditional graphical user interface controls, because that is not its bag. The recommended way to do anything so boring is to embed the sketch inside a larger externally coded application, which is great for people that would rather work in an IDE anyway. But Friday and its Swing setup screen are developed entirely within the PDE (the Spde fork of it), because why crash the party with an IDE for something so simple?

One useful trick for apps with such a setup screen is to return false for displayable in the main sketch class, so the sketch window won’t display until you tell it to. To pop up a separate Swing frame, it’s just a matter of instantiating the object and setting it to visible. Here is some scala-swing code to lay out the first three elements of Friday’s setup frame:

contents = new GridBagPanel {
  add(new Label("NewsGator Account") {
    font = font.deriveFont(java.awt.Font.BOLD)
  }, new Constraints {
    gridwidth = 2
    anchor = Anchor.West
    insets = new Insets(0, 0, 30, 0)
  })
  add(new Label("Don’t have an account?"), new Constraints {
    gridwidth = 2
    gridy = 1
    anchor = Anchor.West
    insets = new Insets(0, 0, 5, 0)
  })
  add(new Button("Sign Up…") {
    reactions += {
      case ButtonClicked(_) => 
        link("http://www.newsgator.com/ngs/order1.aspx")
        name_field.requestFocus()
    }
  }, new Constraints {
    gridwidth = 2
    gridy = 2
    anchor = Anchor.West
    insets = new Insets(0, 0, 30, 0)
  })
  ...

UGHHHHHH. Did you think that a Scala wrapper would make code-based interfaces breezy and fun? Sorry to disappoint! This is probably nicer than the equivalent in regular Java Swing, but it’s still pretty annoying. And boy, wasn’t it fun adjusting each gridy value to mock up different arrangements? (It was not fun.) That process should be done with an interface tool (an “interface builder”, one might call it) that produces resource files. Generating Java code is not the same and is particularly unhelpful in this circumstance where Java is not the language and the the code is not under any IDE’s obsessive control. No, an interface descriptor to go in data/ along with the fonts and images would be a thousand times better.

Anywho, this code-based layout is actually decent looking, even on practically abandoned Java platforms:

So if you have some hours to kill, yes you can cajole Swing layouts into looking passably native. The problem is not the pixels (although they are not perfect or else Quaqua would not exist). The problem is that tweaking layouts is tedious and most programmers won’t bother with it, even with a good layout tool. And without one, nobody is willing to sit through the two hundred tweak-compile-run cycles that a typically complicated interface will require to get right.

Don’t worry though, Friday’s simple setup screen only required fifteen head-smacking cycles.

Runnable jar

Mac app-bundle

Please download in an orderly fashion.

Javascript's Undeclared versus Undefined

2008-12-02T12:05:15+00:00

I learned the hard way that undeclared is different than undefined. If you try to test the simple way for an undeclared variable, you get an error, which in my environment means an annoying pop-up.

/* var a; */  // undeclared
var b;        // undefined

// This would cause an 'object undefined' error
/*
if (a) {
  alert("a is defined");
}
else {
  alert("a is undefined");
}
*/

if (typeof(a) == "undefined") {
  // this will be executed
  alert("a is undeclared or undefined");
}
else {
  alert("a is declared and defined");
}

if (b) {
  alert("b is defined");
}
else {
  // this will be executed
  alert("b is undefined");
}

if (typeof(b) == "undefined") {
  // this will be executed
  alert("b is undeclared or undefined");
}
else {
  alert("b is declared and defined");
}

Coincidence?

2008-12-02T05:23:26+00:00

These two books, long ago separately pre-ordered, arrived from different places on the same day:

One of the things keeping me busy lately has been a Haskell reading group, which has mostly amounted to me teaching some people Haskell, using this book as the main text. We've been using the web preprint, and it's exciting to see the real thing!

For a long time, I've felt that there were lots of decent "first books" on Haskell, but not a single "second book." This is especially unfortunate, given that the most difficult part for many people of learning Haskell is making the jump from CS101 stuff to the actual state of the art. This book rectifies that. In fact, having used it with beginners, I'm not sure I'd recommend it as the best very first book on Haskell, but that hardly matters. Since a lot of this material has until now had to be painstakingly gleaned from mailing list archives, conference papers, functional pearls, blogs and sometimes IRC, this is a very welcome reference. It's exciting to see a Haskell book that goes beyond the basics.

I haven't gotten all the way through it yet, but I'd already happily recommend it. And of course it's great to see the first Scala book in print as well. I understand that 2009 will witness a number of unfortunately similarly named titles. Too bad about the names, otherwise it's a great trend. Congratulations to all six authors of these two books!

Guido's thoughts on Scala

2008-12-01T16:04:09+00:00

Many people who have reacted to Guido's gripes with Scala have bent the post towards a static-typing-is-bad-and-dynamic-typing-is-good syndrome. I think the main point of concern that Guido expresses relates to the complexity of Scala's type system and the many exceptions to the rules for writing idiomatic Scala.

While it is true that there are quite a few rough edges in Scala's syntax today, none of them look insurmountable and will surely be addressed by the language designers in the versions to come. We need to remember that Scala is still a very much growing language and needs time and community feedback to evolve into one of the mainstream general purpose languages. As Martin has mentioned elsewhere, Scala's grammar is no bigger than Java, OCaml or C# and hence the language should not impose a more complex mental model for using the same set of features that the other statically typed languages espouse.

However, Scala's typesystem is expressive enough to offer succinct solutions to many problems that would look much more obtuse in Java. A better modeling of the Expression Problem is a classic example, which, as this paper demonstrates, looks much more verbose, inelegant and non-extensible using Visitors in Java. I have been programming in Scala for sometime now, and I have the feeling that Scala's extremely rich type system can be exploited to implement domain models that encapsulate most of the business rules through declarative type constraints. I have blogged on this very recently and described my experiences of how Scala's type system, specifically, features like abstract types, self type annotations etc., works for you to implement expressive domain models, without writing a single line of runtime type checking code. Less code, more succinct abstractions, and most importantly less tests to write and maintain. After all, everything has a cost, it depends whether you abstract the complexity within the language or within your application.

Another excellent point that has been raised by David Pollak in one of the comments to Guido's post is that we need to judge complexity of language features separately depending on whether you are a library producer or a library consumer. As a library producer, you need to be aware of many of the aspects of Scala's type system, which may not be that essential as a library consumer. Sometime back, I had demonstrated an example of an internal DSL design technique in Scala, where, given a sufficiently typed library for financial trading systems, you can write DSLs in Scala of the form ..

new Order to buy(100 sharesOf "IBM")
  maxUnitPrice 300
  using premiumPricing

Another great example is the Scala parser combinator library, which hides all nuances of Scala's type system and allows you to design external DSLs. Have a look at this example of another financial trading system DSL built using Scala parser combinators ..

On the whole I think Scala has a compelling appeal to the programming community on the JVM. Statically checked duck typing that Scala offers makes your code look and feel like Ruby and yet offers the safety net of compile time checking. Scala is object oriented, though most of idiomatic Scala is functional. In a real world application development environment, where statefulness is the way of life, Scala offers a very good compromise with its hybrid model of programming. Model your states using objects, and model your computations using functional Scala.

Regarding the other concern raised by Guido on the non-uniformity of Scala's syntax, it is indeed true that there are some rough edges to deal with .. like .. () versus {} in for-comprehensions and inconsistent inferring of semi-colons, the number of interpretations that an underscore (_) can have, some subversions in partial function application syntax, syntax for def foo() { .. }. I guess there are quite a few of them that are currently being debated as possible candidates for change. The good part is that the language experts are working with Martin to iron these issues out and smoothen out the rough edges, even at the risk of losing some amount of backwards compatibility.

Do Air Conditioning geeks exist?

2008-12-01T08:27:29+00:00

I work from home now. It rocks I also live in a rental property in the warm climate of Brisbane, Australia.

I wish to convert one of the rooms — westward-facing, 3 metres by 3 metres — into my office. I currently work in the main living area under a split system A/C and this has some problems that I want to alleviate.

Below is an image of the only window in this room. It has a single slider 570mm by 1450mm (the pink line) which is fitted with a security screen by pop-rivets. This window faces west and there is drainage outside.

What is an appropriate set up for cooling such a room, given that it is a rental property? I have looked at evaporative coolers and portable air conditioners but have been advised against both given our climate. It also appears somewhat difficult to use a portable A/C given the fixed security screen and the need for an external vent — can it me reversibly modified somehow? I find no other appropriate option and am a bit of a deadlock. Advice?

Introduction to Automated Proof Verification with SASyLF

2008-12-01T08:00:00+00:00

Doesn’t that title just get the blood pumping? Proof verification has a reputation for being an inordinately academic subject. In fact, even within scholarly (otherwise known as “unrealistically intelligent“) circles, the automated verification of proofs is known mainly as a complex, ugly and difficult task often not worth the effort. This is a shame really, because rigorous proofs are at the very core of both mathematics and computer science. We are nothing without logic (paraphrased contrapositive from Descartes). Believe it or not, understanding basic proof techniques will be of tremendous aid to your cognitive process, even when working on slightly less ethereal problems such as how to get the freakin’ login page to work properly.

Well, if you made it all the way to the second paragraph, then you either believe me when I say that this is legitimately useful (and cool!) stuff, or you’re just plain bored. Either way, read on as we commence our exciting journey into the land of rigorous proofs!

SASyLF Crash Course

If you’re at all familiar with the somewhat-specialized field of proof verification, you probably know that SASyLF (pronounced “sassy elf”) is not the most widely used tool for the job. In fact, it may very well be the least well-known. More commonly, proofs that require automatic verification are written in Twelf or Coq. Both of these are fine tools and capable of a lot more than SASyLF, but they can also be extremely difficult to use. One of the primary motivations behind SASyLF was to produce a tool which was easier to learn, had a higher level syntax (easier to read) and which gave more helpful error messages than Twelf. The main idea behind these convolutions was to produce a tool which was more suitable for use in the classroom.

The main design decision which sets SASyLF apart from Twelf is the way in which proofs are expressed. As I understand it, Twelf exploits Curry-Howard correspondence to represent proofs implicitly in the types of a functional program (disclaimer: I’ve never used Twelf, this is just coming from what little I have read about it). While this can be very powerful, it’s not the most intuitive way to think about a proof. Eschewing this approach, SASyLF expresses proofs using unification (very similar to Prolog) and defines inference rules explicitly in a natural-language style.

There are three main components to a SASyLF proof:

Syntax
Judgments
Theorems/Lemmas

Intuitively enough, the syntax section is where we express the grammar for the language used throughout our proof. This grammar is expressed very naturally using BNF, just as if we were defining the language mathematically for a hand-written proof. Left-recursion is allowed, as is right-recursion, arbitrary symbols, ambiguity and so on. SASyLF’s parser is mind bogglingly powerful, capable of chewing threw just about any syntax you throw at it. The main restriction is that you cannot use parentheses, square brackets ([]), pipes (|) or periods (.) in your syntax. The pure-untyped lambda calculus defined in SASyLF would look something like this:

t ::= fn x => t[x]
    | t t
    | x

I said we couldn’t use brackets, but that’s only because SASyLF assigns some special magic to these operators. In a nutshell, they allow the above definition of lambda calculus to ignore all of the issues associated with variable name freshness and context. For simplicity’s sake, that’s about as far as I’m going to go into these mysterious little thingies.

The judgments section is where we define our inference rules. Just as if we were defining these rules by hand, the syntax has the conditionals above a line of hyphens with the conclusion below. The label for the rule goes to the right of the “line”. What could be more natural?

judgment eval: t -> t

t1 -> t1′
--------------- E-Beta1
t1 t2 -> t1′ t2

The judgment syntax is what defines the syntax for the -> “operator”. Once SASyLF sees this, it knows that we may define rules of the form t -> t, where t is defined by the syntax section. Further on down, SASyLF sees our E-Beta1 rule. Each of the tokens within this rule (aside from ->) begins with “t“. From this, SASyLF is able to infer that we mean “a term as defined previously”. Thus, this rule is syntactically valid according to our evaluation judgment and the syntax given above.

Of course, theorems are where you will find the real meat of any proof (I’m using the word “proof” very loosely to mean the collection of proven theorems and lemmas which indicates some fact(s) about a language). SASyLF wouldn’t be a very complete proof verification system without support for some form of proving. Once again, the syntax is extremely natural language, almost to the point of being overly-verbose. A simple theorem given the rules above plus a little would be to show that values cannot evaluate:

theorem eval-value-implies-contradiction:
    forall e: t -> t’
    forall v: t value
    exists contradiction .

    _: contradiction by unproved
end theorem

Note that contradiction is not more SASyLF magic. We can actually define what it means to have a contradiction by adding the following lines to our judgment section:

judgment absurd: contradiction

In other words, we can have a contradiction, but there are no rules which allow us to get it. In fact, the only way to have a contradiction is to somehow get SASyLF to the point where it sees that there are no cases which satisfy some set of proven facts (given the forall assumptions). If SASyLF cannot find any cases to satisfy some rules, it allows us to derive anything at all, including judgments which have no corresponding rules.

Readers who have yet to fall asleep will notice that I cleverly elided a portion of the “theorem” code snippet. That’s because there wasn’t really a way to prove that contradiction given the drastically abbreviated rules given in earlier samples. Instead of proving anything, I used a special SASyLF justification, unproved, which allows the derivation of any fact given no input (very useful for testing incomplete proofs). Lambda calculus isn’t much more complicated than what I showed, but it does require more than just an application context rule in its evaluation semantics. In order to get a taste for SASyLF’s proof syntax, we’re going to need to look at a much simpler language.

Case Study: Integer Comparison

For this case study, we’re going to be working with simple counting numbers which start with 0 and then proceed upwards, each value expressed as the successor of its previous value. Thus, the logical number 3 would be s s s 0. Not a very useful language in the real world, but much easier to deal with in the field of proof verification. The syntax for our natural numbers looks like this:

n ::= 0
    | s n

With this humble definition for n, we can go on to define the mathematical greater-than comparison using two rules under a single judgment:

judgment gt: n > n

------- gt-one
s n > n

n1 > n2
--------- gt-more
s n1 > n2

Believe it or not, this is all we need to do in terms of definition. The first rule says that the successor of any number is greater than that same number (3 > 2). The second rule states that if we already have two numbers, one greater than the other (12 > 4), then the successor of the greater number will still be greater than the lesser (13 > 4). All very intuitive, but the real question is whether or not we can prove anything with these definitions.

An Easy Lemma

For openers, we can try something reasonably simple: prove that all non-zero numbers are greater than zero. This is such a simple proof that we won’t even bother calling it a theorem, we will give it the lesser rank of “lemma”:

lemma all-gt-zero:
    forall n
    exists s n > 0 .

    _: s n > 0 by induction on n:
        case 0 is
            _: s 0 > 0 by rule gt-one
        end case

        case s n1 is
            g: s n1 > 0 by induction hypothesis on n1
            _: s s n1 > 0 by rule gt-more on g
        end case
    end induction
end lemma

In order to prove anything about n, we first need to “pull it apart” and find out what it’s made of. To do that, we’re going to use induction. We could also use case analysis, but that would only work if our proof didn’t require “recursion” (we’ll get to this in a minute). There are two cases as given by the syntax for n: when n is “0“, and when n is “s n1“, where n1 is some other number. We must prove that s n > 0 for both of these cases individually, otherwise our proof is not valid.

The first case is easy. When n is 0, the proof is trivial using the rule gt-one. Notice that within this case we are no longer proving s n > 0, but rather s 0 > 0. This is the huge win brought by SASyLF’s unification: n is “0” within this case. Anything we already know about n, we also know about 0. When we apply the rule gt-one, SASyLF sees that we are attempting to prove s n > n where n is “0“. This is valid by the rule, so the verification passes.

The second case is where things get interesting. We have that n is actually s n1, but that doesn’t really get us too much closer to proving s s n1 > 0 (remember, unification). Fortunately, we can prove that s n1 > 0 because we’re writing a lemma at this very moment which prove that. This is like writing a function to sum all the values in a list: when the list is empty, the result is trivial; but when the list has contents, we must take the head and then add it to the sum of the tail as computed by…ourself. Induction is literally just recursion in logic. Interestingly enough, SASyLF is smart enough to look at all of the inductive cases in your proof and verify that they are valid. This is sort-of the equivalent of a compiler looking at your code and telling you whether or not it will lead to an infinite loop.

To get that s n1 > 0, we use the induction hypothesis, passing n1 as the “parameter”. However, we’re not quite done yet. We need to prove that s s n1 > 0 in order to unify with our original target (s n > 0). Fortunately, we already have a rule that allows us to prove the successor of a number retains its greater-than status: gt-more.

However, gt-more has a condition in our definition. It requires that we already have some fact n1 > n2 in order to obtain s n1 > n2. In our case, we already have this fact (s n1 > 0), but we need to “pass” it to the rule. SASyLF allows us to do this by giving our facts labels. In this case, we have labeled the s n1 > 0 fact as “g“. We take this fact, pack it up and send it to gt-more and it gives us back our final goal.

A Slightly Harder Theorem

A slightly more difficult task would be to prove that the successors of two numbers preserves their greater-than relationship. Thus, if we know that 4 > 3, we can prove that 5 > 4. More formally:

theorem gt-implies-gt-succ:
    forall g: n1 > n2
    exists s n1 > s n2 .

    _: s n1 > s n2 by unproved
end theorem

At first glance, this looks impossible since we don’t really have a rule dealing with s n on the right-hand side of the >-sign. We can try to prove this one step at a time to see whether or not this intuition is correct.

Almost any lemma of interest is going to require induction, so immediately we jump to inducting on the only fact we have available: g. Note that this is different from what we had in the earlier example. Instead of getting the different syntactic cases, we’re looking at the the rules which would have allowed the input to be constructed. After all, whoever “called” our theorem will have needed to somehow prove that n1 > n2, it would be helpful to know what facts they used to do that. SASyLF allows this using the case rule syntax. We start with the easy base case:

_: s n1 > s n2 by induction on g:
    case rule
        ------------ gt-one
        _: s n2 > n2
    is
        _: s s n2 > s n2 by rule gt-one
    end case
end induction

In this case, the term _: s n2 > n2 is unified with n1 > n2. Thus, n1 is actually “s n2“. This means that by unification, we are actually trying to prove s s n2 > s n2. Fortunately, we have a rule for that. If we let “n” be “s n2“, we can easily apply the rule gt-one to produce the desired result.

The second case is a bit trickier. We start out by defining the case rule according to the inference rules given in the judgment section. The only case left is gt-more, so we mindlessly copy/paste and correct the variables to suit our needs:

case rule
    g1: n11 > n2
    ------------- gt-more
    _: s n11 > n2
is
    _: s s n11 > s n2 by unproved
end case

In this case, n1 actually unifies with “s n11“. This is probably the most annoying aspect of SASyLF: all of the syntax is determined by token prefix, so every number has to start with n, occasionally making proofs a little difficult to follow.

At this point, we need to derive s s n11 > s n2. Since the left and right side of the > “operator” do not share a common sub-term, the only rule which could possibly help us is gt-more. In order to apply this rule, we will somehow need to derive s n11 > s n2 (remember, gt-more takes a known greater-than relationship and then tells us something about how the left-successor relates to the right). We can reflect this “bottom-up” step towards a proof in the following way:

case rule
    g1: n11 > n2
    ------------- gt-more
    _: s n11 > n2
is
    g: s n11 > s n2 by unproved
    _: s s n11 > s n2 by rule gt-more on g
end case

At this point, SASyLF will warn us about the unproved, but it will happily pass the rest of our theorem. This technique for proof development is extremely handy in more complicated theorems. The ability to find out whether or not your logic is sound even before it is complete can be very reassuring (in this way you can avoid chasing entirely down the wrong logical path).

In order to make this whole thing work, we need to somehow prove s n11 > s n2. Fortunately, we just so happen to be working on a theorem which could prove this if we could supply n11 > n2. This fact is conveniently available with the label of “g1“. We feed this into the induction hypothesis to achieve our goal. The final theorem looks like this:

theorem gt-implies-gt-succ:
    forall g: n1 > n2
    exists s n1 > s n2 .

    _: s n1 > s n2 by induction on g:
        case rule
            ------------ gt-one
            _: s n2 > n2
        is
            _: s s n2 > s n2 by rule gt-one
        end case

        case rule
            g1: n11 > n2
            ------------- gt-more
            _: s n11 > n2
        is
            g2: s n11 > s n2 by induction hypothesis on g1
            _: s s n11 > s n2 by rule gt-more on g2
        end case
    end induction
end theorem

Conclusion

I realize this was a bit of a deviation from my normal semi-practical posts, but I think it was still a journey well worth taking. If you’re working as a serious developer in this industry, I strongly suggest that you find yourself a good formal language and/or type theory textbook (might I recommend?) and follow it through the best that you can. The understanding of how languages are formally constructed and the mental circuits to create those proofs yourself will have a surprisingly powerful impact on your day-to-day programming. Knowing how the properties of a language are proven provides tremendous illumination into why that language is the way it is and somtimes how it can be made better.

Credit: Examples in this post drawn rather unimaginatively from Dr. John Boyland’s excellent course in type theory.

SASyLF Main Page at Carnegie Mellon University
Download math.slf (the full greater-than example)
Further reading…

ObserveFunctorMonad

2008-11-30T08:58:51+00:00

An exercise arising from an IRC discussion in #scala:

trait Functor[F[_]] {
  def fmap[A, B](fa: F[A], f: A => B): F[B]
}
 
trait Monad[M[_]] {
  def bind[A, B](ma: M[A], f: A => M[B]): M[B]
  def pure[A](a: A): M[A]
}
 
object ObserveFunctorMonad {
  def observe[K[_]](m: Monad[K]): Functor[K] = error("your homework")
}

Ubuntu released 8.10 knowing there was a nasty defect

2008-11-30T04:29:29+00:00

I had a rant about Ubuntu 8.10 being a buggy release. Turns out that the Canonical people knew about the defect and release Intrepid anyway. The problem impacts about 10% of modern laptops... ones with the most common Intel WiFi chipset. The defect has not been fixed. The workaround doesn't work. What are these people thinking?

Why Functional Programming Matters in short prose

2008-11-30T01:35:51+00:00

Why Functional Programming Matters paraphrased — a result of a discussion in an IRC channel. Others may find value.

A program may be represented as a function that accepts some input and produces some outputλ This function may be composed of parts; other functions and values.

Suppose any function that exists. I will suppose one that is made of parts A, B, C and D. I might denote this A ∘ B ∘ C ∘ D. Now suppose you desire a function that is made of parts A, B, C and E. Then the effort required for your function should be proportional to the effort required for E alone.

The closer you are to achieving this objective for any function, the closer you are to the essence of (pure) functional programming. This exists independently of the programming language, though programming languages will vary significantly in the degree to which they accommodate this objective.

Imperative programming — specifically the scope of a destructive update — is the anti-thesis of this objective.

Some Wicke[dt] Scala

2008-11-29T18:45:27+00:00

Apache Wicket is a cool web framework to develop with. I’m in that business for some time now and I still haven’t found any nicer - really! JSF is no match at all, and since I’m doing most of my programming in Scala I’d taken a brief look at Lift. But that’s not mine, either. Everything in Wicket is ad-hoc reusable, that’s what I like best about it. And I think of Wicket and Scala as the perfect combination to develop with: Scala’s traits allow adding and overriding things so easy to Wicket components that it’s just a joy!

I want to add some foundations to above statements, so be prepared for some tough Wicket internal stuff - easily written in Scala! And for the readers still developing Wicket applications in Java: You might find the resulting library equally useful and should be able to use it from Java as well.

Enter the Quest

You may have found yourself in need to preprocess the markup for your Wicket components. Typically, you like using FreeMarker or Velocity for that. There are Wicket projects which provide you with some kind of support to do so. But those only go half the way. You can load template files and process them, but you can’t add Wicket components in them! Actually you could using <wicket:component> - but that’s a) not officially supported and b) doesn’t work when adding components with Ajax behaviors, since those components are coming in in a late stage in Wicket’s rendering scheme. Suppose you want to add some self-updating RSS feed panels based on some markup coming from a database you’re lost here.

The solution: Dynamo

I had such a task and found myself writing a generalized template engine with support for dynamic Wicket components. Here’s a short code example how it’s working:

^?View Code SCALA

class FreemarkerApplication extends WebApplication with Dynamo {
  def getTemplateEngine: TemplateEngine = new FreemarkerEngine( appPath, new Properties, Locale.GERMAN )
  def getHomePage = classOf[FreemarkerDynamoPage]
}
 
class FreemarkerPage extends WebPage with TemplateContainer {
  val getTemplateName = "DynamoPage.ftl"
  val getParameterMap = new java.util.Map[String,AnyRef]
}
 
class FreemarkerDynamoPage extends FreemarkerPage with DynamicComponentContainer

That’s it! All logic is hidden behind the scenes, you just have to know it works. Your dynamic code might add components in the following way:

^?View Code HTML

    <h3>Dynamic Wicket Components:</h3>
    <#list StringList as s>
      <dynComponent
        class="de.footprint.wicket.application.dynamo.TextPanel"
        label="${s}" />
    </#list>

But how?

There were three tasks to accomplish. First was to define a general interface to call the processed template file. That wasn’t too difficult:

^?View Code SCALA

trait TemplateEngine {
  @throws( classOf[ResourceStreamNotFoundException] )
  def processTemplate( templateName: String, parameter: java.util.Map[String,AnyRef], locale: Locale ): String
 
  def resetCache: Unit
}

Second was to hook into the markup loader to call on the engine whenever a container of type TemplateContainer was required. The Dynamo trait registers a customzed MarkupCache with the Application which just does that.

The third task was to add the dynamic component stuff. Now it got a bit nasty. The resource stream had to be filtered for components added by the template engine - and that had to happen before Wicket got it’s hand on the resources, i.e. WicketTagIdentifier got on the resource. So, the markup had to be loaded manually beforehand and the additional components had to be given unique (and stable) wicket:id’s. Having all that, the components only had to be created and added with their correct wicket:id. Easy, huh?

The nice thing with Scala is, you can just hide all these tasks in traits and add them to the components. onBeforeRender will just be overriden and mixed in, so the functionality is magically just there.

Famous last words

The code is here: Dynamo

To run the examples you have to use mvn resin:run.

It includes general support for FreeMarker and Velocity, but this approach can easily be integrated with any other markup source provider.

I hope you find this useful and get some ideas how Scala/Wicket can boost your code - not only in function but also in readability.

— Jan.

Mount Mee State Forest Campgrounds

2008-11-29T10:52:51+00:00

I am an avid GPS user and I visit Mount Mee State Forest regularly on my dirt bike (Husqvarna TE450). I encounter queries on forums for the locations of the campgrounds in this forest, so here they are:

Neurum Creek Campground
-27.061853° 152.696228°
-27°03′42.67″ 152°41′46.42″

Archer Campground
-27.016317° 152.697767°
-27°00′58.74″ 152°41′51.96″

You can get to Neurum Creek Campground by taking Sellin Road then turning onto Neurum Creek Road and following it north. Archer Campground can be found by taking Sellin Road then Lovedays Road, however, it is easier to approach from the north side of the forest and head south. Both of these campgrounds require a permit through the Environmental Protection Agency (EPA) or you can book online.

There is another campground called Neurum Creek Bush Retreat but this is a different campground to those two mentioned. It is privately owned and I have personally stayed a weekend there and recommend it, especially if you have children or want a weekend of dirt bike riding (though the operators are understandably strict about noise control) since you can head south into Mount Mee State Forest or head over to Beerburrum State Forest. This campground is even further north of the state forest than the other two (which are situated in the forest boundaries), but is a short ride on gravel to the forest entrance.

Here is a reasonably comprehensive GPX file of Mount Mee State Forest including the two campgrounds and many dirt trails. I’d upload it to OpenStreetMap but I have yet to figure that out.

I use a Garmin 60Cx unit.

I hope this helps

scalaimage2

2008-11-29T02:23:10+00:00

With my growing frustration with Matlab, I’ve been looking for a while for a language that was

Garbage collected
Good notation for numerical computation
Fast enough for numerical computation

After a long search, I think I’ve finally found my home in Scala. Today I did a very trivial, self-contained computation of some images of the Mandelbrot set to test it out.

When learning Scala, I couldn’t find enough examples of numerical stuff, so I thought I would post this here for those potentially interested. Here’s a first attempt:

import java.io._
import java.lang.Math

object mandelbrot {
  // tiny complex number class including syntactic sugar for basic operations
  class Complex(val a: Double, val b: Double){
    // represents the complex number a + b*i
    def +(that: Complex) = new Complex(this.a+that.a,this.b+that.b)
    def *(that: Complex) = new Complex(this.a*that.a-this.b*that.b,this.a*that.b+that.a*this.b)
    def abs() = Math.sqrt(this.a*this.a + this.b*this.b)
  }

  def run(n: Int, level: Int) : Unit = {
    val out = new FileOutputStream("scalaimage.pgm")
    out.write(("P5\n"+n+" "+n+"\n255\n").getBytes())

    for {y0 <- 0 until n
	 x0 <- 0 until n }{
	   // y0 and x0 are in pixel integers
	   // x  and y  are real number coordinates
	   val x = -2.0 + x0*3.0/n
	   val y = -1.5 + y0*3.0/n

	   var z = new Complex(0,0)
	   var c = new Complex(x,y)
	   for(i <- 0 until level)
	     z = z*z + c 

	   if (z.abs < 2)
	     out.write(0);
	   else
	     out.write(255);
	 }
    out.close()
  }

  def main(args: Array[String]) {
    run(Integer.parseInt(args(1)), Integer.parseInt(args(0)))
  }
}

Not bad, right? Notice how painlessly we create the Complex class, along with the natural syntax for manipulating the numbers. What’s more, the result is very fast. For a 2048 by 2048 image, I get results in less than 30 seconds. (Just consider running this algorithm in Matlab. Ha!)

Save this to a file “mandelbrot1.scala”, and compile and run with:

scalac mandelbrot1.scala
scala mandelbrot 100 2048

The first argument is many times to try the mandelbrot iteration, and the second argument is how big of an image to output. The result is:

(I manually converted the .pgm to .png to save space– click to get the full resolution version.)

A slightly more complex version follows. I added notation for the operators *= and +=, which speeds things up by about 10%. I also now color the pixels by how many iterations it takes the mandelbrot iterations to diverge.

import java.io._
import java.lang.Math

object mandelbrot {
  // tiny complex number class including syntactic sugar for basic operations
  class Complex(var a: Double, var b: Double){
    // represents the complex number a + b*i
    def +(that: Complex) = new Complex(this.a+that.a,this.b+that.b)
    def *(that: Complex) = new Complex(this.a*that.a-this.b*that.b,this.a*that.b+that.a*this.b)
    def abs() = Math.sqrt(this.a*this.a + this.b*this.b)
    def *=(that: Complex) ={
      val newa = this.a*that.a-this.b*that.b
      this.b = this.a*that.b+that.a*this.b
      this.a = newa
      this
    }
    def +=(that: Complex)={
      this.a += that.a
      this.b += that.b
      this
    }
  }

  def run(n: Int, level: Int) : Unit = {
    val out = new FileOutputStream("scalaimage.pgm")
    out.write(("P5\n"+n+" "+n+"\n255\n").getBytes())

    for {y0 <- 0 until n
	 x0 <- 0 until n }{
	   // y0 and x0 are in pixel integers
	   // x  and y  are real number coordinates
	   val x = -2.0 + x0*3.0/n
	   val y = -1.5 + y0*3.0/n

	   var z = new Complex(0,0)
	   var c = new Complex(x,y)
	   var i = 0
	   do {
	     z *= z
	     z += c
	     i += 1
	   } while( z.abs < 2 && i < level)

	   if (z.abs < 2)
	     out.write(0);
	   else
	     out.write( (i*255.0/level).toInt );
	 }
    out.close()
  }

  def main(args: Array[String]){
    run(Integer.parseInt(args(1)), Integer.parseInt(args(0)))
  }
}

Compile and run as before. The output is:

I’m not sure how many people are using Scala for numerical applications, but it looks very good so far. There are a few minor tricks that turn out to be incredibly convenient. For example, if you define any function func for some class a, in a normal language you would call it by

a.func(b).

Scala defines this, but also gives you automatically the notation

a func b.

Adding in Scala’s non-mandatory type declarations (as above, notice that x0 and x aren’t declared to be Int or Double) tricks for implicit conversion between types, and it looks like it is possible to recreate almost all of Matlab’s synactic sugar for matrix manipulation, while actually allowing reasonable speed for non-vectorizable code, and reasonable facilities for abstraction. I hope to soon be an ex-Matlab user.

The scalala library looks very promising, though I haven’t taken the time to install it yet, and it looks like absolutely no one is using it. (dramage, give us some documentation!)