Planet Scala

Nikolaj Lindberg: Book: Real World Haskell (not much real world so far :)

noreply@blogger.com (Nikolaj Lindberg) — Wed, 07 Jan 2009 02:00:18 +0000

I've just started to read Real World Haskell (the paper book). It seems like a nice book (except for a few irritating and confusing typos/mistakes at the start of the book).

However, I've read more than 100 pages so far, and still not a sign of any of the "real world" stuff promised by the title. I still don't know much or anything about IDE:s, how to compile the code, scripting, any practical details on how to structure your code into modules, or anything in that direction. So far, mostly (sometimes rather long-wined) discussions on specific (list) functions. One of the examples, end up in a conclusion that might be paraphrased as "by the way, don't use the function we've discussed the last few pages; in real world settings it doesn't work too well".

In the real world, you run into both needles and haystacks , occasionally, but that doesn't help making sense of

isInAny3 needle haystack = any (isInfixOf needle) haystack

And one more real world example of the kind zip3foobar "quux" and I may start losing interest... or just start screaming.

Well, the upcoming chapters have promising names, so I guess I've just have to keep on reading. And I guess you have to start with the basics. Still, over 100 pages, and mostly foobars so far...

The book is available on-line.

Eric Torreborre: Doing my homework

noreply@blogger.com (Eric) — Wed, 07 Jan 2009 00:40:47 +0000

The mapFilter function

I recently tried to write a function using Scala which would apply a function to a list, then filter all the resulting elements according to a criteria. In my infinite wisdom I named it "mapFilter". Here is an example of what I expected:

mapFilter(List(1, 2), (x:Int) => if (x%2 == 0) Some(x*2) else None)
// returns List(4)

Of course I wanted this function to be as efficient as possible so it had to do the job in one pass. Otherwise, I could just use a combination of "map" and "filter" to do the trick:

def mapFilter[T, U](l: List[T], f: T =>Option[U]): List[U] = {

l.map(f(_)).filter(_.isDefined).map(_.get)

3 passes on the list, we can definitely do better than this!

Using flatMap

Somehow I remembered that the Option class is a Monad, and my subconscious related the bind operation of monads to the flatMap operation on Iterable. I should be able to do something with it,...

Then I experimented this:

List(1, 2) flatMap { (x:Int) => if (x%2 == 0) Some(x*2) else None }

Bingo, it works!

Now, if you read the signature of the flatMap method (on the Iterable trait), understanding how this works is not so clear:

def flatMap[B](f: A => Iterable[B]): Iterable[B]

Option is not an Iterable, this shouldn't compile! Yes, it does, because any Option can be implicitly converted to an Iterable thanks an implicit conversion:


object Option {
  /** An implicit conversion that converts an option to an iterable value */
  implicit def option2Iterable[A](xo: Option[A]): Iterable[A] = xo.toList

Sometimes I wish there was a way to display all the available implicit definitions for a given scope,...

Monads again,...

What about the "Monadic" stuff then? Tony Morris, on the Scala mailing-list, left me a message hinting at a purer solution:

Hi Eric,
You can write mapFilter using Scalaz which has a MonadEmptyPlus data
structure. I suggest you do not use the available subversion of the
List.flatMap type signature from which to learn

So I thought that I should do my homework, have a look at it and understand why MonadEmptyPlus was a good way to write the mapFilter function.

You can find all the code from the scalaz project here. The package I'm exploring today is the "control" package.

A bit of Scalaz exploration

Let's jump right to the MonadEmptyPlus:


trait MonadEmptyPlus[M[_]] extends MonadPlus[M] with MonadEmpty[M]

Ohhh. We are high on abstraction here, because MonadPlus is:

trait MonadPlus[M[_]] extends Monad[M] with Plus[M]

And MonadEmpty is, as you can guess:

trait MonadEmpty[M[_]] extends Monad[M] with Empty[M]

Last but not least, Monad is:

trait Monad[M[_]] extends Pure[M] with Bind[M] with Functor[M] with Apply[M]

Honestly, I didn't suspect that my homework would take me so long! Yet, since every concept and notion of what is a Monad seems to be carefully crafted in Scalaz, this is a good idea to spend a bit of time trying to understand each of them.

Pure

The Pure trait is defined by:


/**
 * Project the given value into the environment that is abstracted over. 
 */
def pure[A](a: A): P[A]

It actually says that you can take any value of some type A, and using the "pure" function, put it in a "Container" of type P, called the "environment" here. I guess that the idea is to say that once the value is well protected and controlled in its container, it is "pure". That being said, this is still very abstract. Fortunately we have some implicit values in the the Pure object giving us some examples of what it means to be "pure":

 implicit val OptionPure = new Pure[Option] {
  def pure[A](a: A) = Some(a)
}
implicit val ListPure = new Pure[List] {
  def pure[A](a: A) = List(a)
}

A Pure[Option] of the value a is simply the value "a" inside the "Some" container. So if you think of Monads as "Containers" for computation then "pure" is what creates the container with a value inside.

Bind

Bind is the evil twin of Pure and, as far as I'm concerned, the essence of monads because it is the basis of computing with Monads:


/**
 * Binds a function through an environment (known also as sequencing).
 */
trait Bind[BD[_]] {
  /**
   * Binds the given value with the given value through the environment.
   */
  def bind[A, B](f: A => BD[B], ba: BD[A]): BD[B]
}

Every word is important here. You have an "environment" again. This "environment" or "Container" contains values which have been put there using the pure function for example. Now, what you want to do is to apply a function to the value inside the container, without the value leaving it. So you "bind" a function to the environment. Again, concrete examples, given by the implicit values in the Bind object will help us understand what's going on:

implicit val OptionBind = new Bind[Option] {
  def bind[A, B](f: A => Option[B], a: Option[A]) = a flatMap f
}
implicit val ListBind = new Bind[List] {
  def bind[A, B](f: A => List[B], a: List[A]) = a flatMap f
}

When I bind a function to an Option, like Some(4), it is as if I'm doing:


val f = x => Some(x + 2)

bind(f, Some(4))

1. apply the function to the inner element

Some(f(4)) === Some(Some(6))

2. "flatten" the inner result by removing the inner "box"

Some(6)

This brings 2 remarks here:

1. "bind" for a monad is indeed the "flatMap" function of an Iterable in Scala. It maps the value then flatten the results. The big difference is in the function signature. While flatMap is a method on Iterable and accepts a function returning another Iterable, the bind function accepts a "Container" type of any sort and returns the same container type; Binding to an Option will return an Option, binding to a List will return a List ("I suggest you do not use the available subversion of the List.flatMap type signature from which to learn"...)

Functor

2. why is it necessary to have a "bind" operation? If I want to get Some(6) as a result, something like "map" should be enough. True. And this even has a name, this is the "fmap" operation of the Functor trait (also mixed in by Monad):


trait Functor[F[_]] {
 /**
  * Maps the given function across the given environment.
  */
  def fmap[A, B](f: A => B, fa: F[A]): F[B]
}

The "fmap" operation just means transforming the value(s) inside the Container to something else. But the bind operation offers an additional advantage. It allows to compose functions returning values in the Container. I can't compose directly 2 functions returning an Option:

val f = (x:Int) => if (x % 2 == 0) Some(x) else None
val g = (x:Int) => if (x % 3 == 0) Some(x) else None

// this doesn't work since g returns an Option and f expects an Int
val composed = f . g // ??

But I can compose them using "bind":

// this works as expected
val composed: (Int => Option[Int]) = (x:Int) => g(x).bind(f)

Apply

The definition of Apply is this one:


trait Apply[AP[_]] {
  def apply[A, B](f: AP[A => B], fa: AP[A]): AP[B]
}

And a good example is provided for Lists:


implicit val ListApply = new Apply[List] {
  def apply[A, B](f: List[A => B], a: List[A]) = f flatMap (f => a map(f(_)))

We take a list of functions, a list of values, and we return a list with the results of all the functions applied to all the values. Actually this is the essence of the "Applicative" model of programming, so I don't really understand why it's been added to the Monad trait. I guess it is because monad computations imply applicative computations as described here.

Empty and Plus

Now that the tour of Monad is over, we can look at the Empty and Plus traits.

Empty is very easy, it just returns an empty "Container" (named "Environment" here):

trait Empty[E[_]] {
  /**
   * Returns the empty environment.
   */
  def empty[A]: E[A]
}

implicit val OptionEmpty = new Empty[Option] {
  def empty[A] = None
}

implicit val ListEmpty = new Empty[List] {
  def empty[A] = Nil
}

Plus defines what it means to "append" or "merge" two environments together. Its result is very specific to the underlying Container type as you can see with the implicit values:


trait Plus[P[_]] {
  /**
   * Appends the two given values.
   */
  def plus[A](a1: => P[A], a2: => P[A]): P[A]
}

implicit val OptionPlus = new Plus[Option] {
  def plus[A](a1: => Option[A], a2: => Option[A]) = a1 orElse a2
}

implicit val ListPlus = new Plus[List] {
  def plus[A](a1: => List[A], a2: => List[A]) = a1 ::: a2
}

Empty and Plus are indeed defining some kind of addition operation on Monads, aren't they?

The real academic mapFilter function ;-)

With all that at hand I should now be able to create my mapFilter function ;-)


import scalaz.control._
def mapFilter[T, A[_], U](f: T => A[U], m: A[T])(implicit m1: MonadEmptyPlus[A])= {

  m1.bind(f, m)
}

Is that all? Yes, let's try it:


// Note that Scala type inferencer doesn't infer the types here
mapFilter[Int, List, Int]((x:Int) => if (x%2 == 0) List(x*2) else Nil,
                          List(1, 2))

However we may wonder: is that really necessary to have a MonadEmptyPlus? No, a simple Monad is also ok:


def mapFilter[T, A[_], U](f: T => A[U], m: A[T])(implicit m1: Monad[A]) = {

m1.bind(f, m)

And that's understable with the List example because we're using "flatMap" under the covers! But this is not totally satisfying. Somehow, I would like to enforce that if f returns the empty element for a given MonadEmptyPlus, then that element is filtered from the result.

Even better with FoldRight

We can actually define this using another abstraction from the control package: FoldRight.


trait FoldRight[F[_]] {
  /**
   * Fold the given function across the given environment.
   */
  def foldRight[A, B](t: F[A], b: B, f: (A, => B) => B): B
}

FoldRight works exactly as the foldRight method on Iterable in Scala and allows us to use the Empty and Plus definition from MonadEmptyPlus, accumulating the mapped values, unless they're empty:


def mapFilter[T, A[_], U](m: A[T], f: T => A[U])
                         (implicit m1: MonadEmptyPlus[A], f1: FoldRight[A]) = {
  f1.foldRight[T, A[U]](m, m1.empty, (t, result) => m1.plus(f(t), result))
}

And that's the end of the assignement for today!

Conclusion

This exploration of Scalaz was very interesting for me because:

The abstractions are nicely separated
Implicit values provide lots of concrete examples
The programming style is very close to the use of type classes in Haskell (which wraps my mind in a new way). The drawback is less type inference by Scala
I had to think again about that "bind" function and why it was so special
It gives me the desire to understand more deeply what are the differences between the so-called "models of programming": functor, monads, applicative, arrows,...
I haven't been blogging for a looooonnnnng time.

Joakim Ohlrogge: SWT scala screenshot

Tue, 06 Jan 2009 20:48:48 +0000

At Agile 2008 I was calmly coding together with Daniel Brolund in the open space area. We were interrupted by a guy that asked us if we did any coding in Ruby and if we did he will hold a lightning-talk about a framework he had put together to create SWT-gui:s using JRuby. Interestingly enough I had thought about how GUI-coding could probably be improved in a language such as Ruby so I reluctantly stopped what I was doing and watched the lightning talk. This was the first time I ever saw Glimmer and it totally blew me away!

It must be possible to do in Scala!

I had already started playing with Scala and I thought it would be a cool experiment to see if I could do something similar in Scala. After all, scala is a pretty expressive language.

I made the first attempt a few months ago and I think that qualifies as a failure. I got something to work but all the mutable state and redundant classes I had to create bothered me big time.

I showed what I did to colleague and I got some pointers on how to make my code more function-oriented and hence get rid of some redundancy and mutable state. But my lacking knowledge of Scala and FP prevented me from finishing the task back then.

Success!

Yesterday I gave it another shot and it took me only an evening to make this code:


        shell(
            text("User Profile"),
            composite (
                gridLayout(2, false),
                group (
                    text("Name"),
                    gridLayout(2, false),
                    gridData(FILL, FILL, true, true),
                    label(text("First")), edit(text("Bullet")),
                    label(text("Last")), edit(text("Tooth"))
                ),
                group (
                    gridData(FILL, FILL, true, true),
                    text("Gender"),
                    radioButton (text ("Male"), selected),
                    radioButton (text ("Female"))
                ),
                group (
                    gridData(FILL, FILL, true, true),
                    text("Role"),
                    checkBox (text("Student"), selected),
                    checkBox (text("Employee"), selected)
                ),
                group (
                    text("Experience"),
                    rowLayout,
                    gridData(FILL, FILL, true, true),
                    spinner(selection(5)), label {text ("years")}
                ),
                button (
                    text("save"),
                    gridData(RIGHT, CENTER, true, true)
                ),
                button (
                    text("close"),
                    gridData(LEFT, CENTER, true, true)
                )
           )
        )

Display this:

Pretty cool! I will admit that I have gracefully stolen most of the ideas from Glimmer. Even the example above is a slightly modified Glimmer-example. I will probably go over it a couple of times to see what I can improve from a Scala perspective. I particulary dislike the gridData functions. I think there is room for improvement there.

How does it work?

The general pattern is simple. Take a look a the group element in the code above. Group is implemented like this:


def group(setups:(Group => Unit)*)(parent:Composite) = {
    val group = new Group(parent, SWT.NONE)
    rowLayout(group);
    setups.foreach(setup => setup(group))
}

If you, like me when I saw my first Scala examples, have some dificulty with the double parameterlist notation it is a shorthand in Scala for returning a function. So the above is a function that takes a variable number of functions that take a Group as the only parameter. Unit in Scala is equivalent to void in java. The group-function will return a function that takes a Composite, parent, as it’s only parameter.

The cool thing about this is that you can pass the first set of parameters (the variable number of functions that takes a Group and return Unit) and save the returned function for later evaluation. When the composite parent is later passed what will happen is the following:

A new Group widget will be created with parent as parent
as a default the layout of the group will be set to row layout, this may be overridden in the next step.
Each function in setup will be applied to the newly created group.

Remember, the setup-functions take a group as a parameter so they can perform whatever action on the Group. You have full freedom to do whatever can be done with a group in any of those functions.

The variable length argument is one of the missing pieces I needed to solve the puzzle, I failed to find information about variable arguments via google. Ok, I did not spend an awful lot of time trying…

What does one of those setup-methods look like then?

Let’s start with something simple, the label:


 def label(setups:(Label => Unit)*)(parent:Composite) = {
    val label = new Label(parent, SWT.NONE)
    setups.foreach(setup => setup(label))
}

The pattern repeats itself, lazy creation of the label onto the parent composite (for instance a Group) and the posibility to furter customize the label. Now comes the next secret I had to learn in order to solve the puzzle. Take a look at the text-function:


def text[T <: {def setText(txt:String)}](txt:String)(subject:T) = {
    subject.setText(txt)
}

Pretty straight forward right? Look again at the generic type T. I don’t know what that construct is called, I call it declarative duck-typing, you know if it walks like a duck and quacks like a duck it’s a duck!

Here I declare that T can be any type that has a function named setText that takes a String as an argument and returns nothing. If I did not declare this then the subject.setText(txt) line would not compile. But I’m getting ahead of myself. Why did I need to find this construct?

Several different widgets in SWT has a setText-method, for instance, shell has a setText, label has a setText etc but some controls do not. There is no common interface between all different widgets that has a setText-method. That makes things a bit tricky, my first attempt was to overload different setText-functions, one for label, one for shell etc… This was not only tedious, it didn’t work since the difference between all of those overloaded methods is in the parameterlist of the returned function. Luckily, this little inconvenience forced me to discover this NEAT feature of the Scala-type system. I’m happy I did.

In conclusion

I was able to make a DSL that is very comparable to glimmer which was my original goal. I was able to make the whole thing initialize lazily. Basically I just build a tree of constructing functions and then I sort of light the fuse by passing a shell to the root of that tree and the GUI is created. I did not have to introduce any mutable state other than what is already mutable in SWT. I have a feeling that this approach is very extendable. Maybe the GUI is a good place to introduce Scala in Java-projects? My feeling is that Scala would slowly eat it’s way in to other parts of the system from there

Perhaps this is a good candidate for an open source project? Would you want to use it?

Nikolaj Lindberg: Scala for small throw-away scripting tasks

noreply@blogger.com (Nikolaj Lindberg) — Tue, 06 Jan 2009 16:51:27 +0000

I've come to use Scala for tiny scripts to be thrown away after doing some small task. Typically this involves processing a few files, comparing some textual data, maybe extracting some fields of tab-separated files, etc. The kind of things that Perl used to be the obvious choice for.

Although lacking Perl's simplified syntax for iterating over all lines in files, Scala works quite nicely for small tasks.

For example, today I had to extract from a file all lines of four or more characters including only upper-case characters, and capitalize the output:

scala.io.Source.fromFile(args(0))
.getLines.map(_.stripLineEnd).filter(_.matches("[A-Z]{4,}"))
.map(_.toLowerCase.capitalize).foreach(println)

Not exactly a thing of beauty, but it only took a minute and it works. And it reminds me a bit of a classic Unix command line pipeline.

A few things on my wish-list to make Scala even better for small scripts:

A nicer way of setting the output character encoding (currently you have to do something like Console.setOut(new java.io.PrintStream(Console.out,true,"UTF8")))
It would be great if Source.getLines could remove the new line character of each line
A better name for RichString.stripLineEnd (for some reason, it is totally impossible for me to remember the name of this method)
Maybe scripting support in the Scala Netbeans plugin? (Currently, I think the plugin wants you to put your code in a class/object)

Ruminations of a Programmer: Higher Order abstractions in Scala with Type Constructor Polymorphism

ghosh.debasish@gmail.com (Debasish) — Mon, 05 Jan 2009 12:25:08 +0000

Abstractions at a higher level through type constructor polymorphism. Good type systems are expressive enough to conceal the implementation complexity, and expose *only* what it matters to the developer. People often cringe about the complexity of Scala's type system and how it serves as a barrier to the entry point in mainstream programming. As Michael Feathers recently noted on Tweeter, the unfortunate fact is that people often jump at the esoteric parts of a language before looking at the simpler subset, which he will be using 90% of the time. And, I think Scala has that sweet spot, where you need not mess around too much with variances, implicits and existentials and yet come up with a nice, concise and functional codebase.

In this post, I discuss the already introduced intimidating phrase "Type Constructor Polymorphism" through a series of code snippets ranging from toys to some real-world stuff. The idea is, once again, not to evangelize type theory intricacies, but share some of the experiences of how this feature in Scala's type system can help write idiomatic code, while staying away from the complexities of its underlying implementation.

Jump on ..

We have a list of Option[String] that we need to abstract over and compute some value. Say, for the sake of keeping the example simple, we will calculate the sum of lengths of all the strings present ..

val employees: List[Option[String]] =
  List(Some("dave"), None, Some("john"), Some("sam"))

val n: List[Int] =
  employees.map { x =>
    x match {
      case Some(name) => name.length
      case None => 0
    }
  }.elements.reduceLeft[Int](_ + _)
println(n)

Let us take another problem that needs to abstract over a different List structure, a List of List of Strings, and compute the same result as before, i.e. the sum of lengths of all the strings encountered in the collection ..

val brs: List[List[String]] =
  List(List("dave", "john", "sam"), List("peter", "robin", "david"), List("pras", "srim"))

val m: List[Int] =
  brs.flatMap {x => x.map {_.length}}
     .elements.reduceLeft[Int](_ + _)
println(m)

Do you see any commonality in the solution structure in the above snippets ? After all, the problem space has a common generic structure ..

we have a List with some String values abstracted in different forms

need to iterate over the list

do some stuff with elements in the list and compute an Int value

Unfortunately the actual solution structures look quite different and have to deal a lot digging into the abstractions of the underlying representations within the collection itself. And this is because, we cannot abstract over the type constructor (the List in this case) that takes another type constructor as an argument (Option[String] in the first case and List[String] in the second case).

Enter type constructor polymorphism.

Sounds intimidating ? Maybe, but ask the Haskellers .. they have been using typeclasses ever since so successfully in comprehensions, parser combinators and embedded DSLs and programming at a different level of abstraction.

Scala supports type constructor polymorphism since 2.5, and the details have been discussed in a recent paper by Adriaan Moors et al in OOPSLA 2008.

Here is a snippet of the Scala code that works seamlessly for both of the above cases ..

val l: List[Int] = employees.flatMapTo[Int, List]{_.map{_.length}}
val sum: Int = l.elements.reduceLeft[Int](_ + _)
println(sum)

The above code abstracts over List through higher order parametric polymorphism, i.e. independent of whether the List parameter is an Option[] or another List[]. Incidentally both of them (List and Option) are monads and flatMapTo abstracts a monadic computation, hiding all details of type constructor polymorphism from the developer.

Now here is some real life example (elided for simplicity) ..

Here are the simple domain models for Customer, Instrument and Trade, used for modeling a use case where a Customer can order for the Trade of an Instrument in an exchange.

case class Customer(id: Int, name: String)
case object nomura extends Customer(1, "NOMURA")
case object meryll extends Customer(2, "MERYLL")
case object lehman extends Customer(3, "LEHMAN")

case class Instrument(id: Int)
case object ibm extends Instrument(1)
case object sun extends Instrument(2)
case object google extends Instrument(3)

case class Trade(ref: Int, ins: Instrument, qty: Int, price: Int)

And we fetch the following list through a query from the database. It is a List of tuples where each tuple consists of a Customer and a trade that has been executed based on the Order he placed at the exchange. And here is the snippet of the code that computes the sum total of the values of all trades executed in the day for all customers.

val trades: List[(Customer, Option[Trade])] =
  List((nomura, Some(Trade(100, ibm, 20, 12))),
       (meryll, None), (lehman, Some(Trade(200, google, 10, 10))))

val ts: List[Option[Trade]] = trades.map(_._2)
val t: List[Int] = ts.flatMapTo[Int, List]{_.map{x => x.qty * x.price}}
val value = t.elements.reduceLeft[Int](_ + _)
println(value)

Really not much different from the above simple cases where we were dealing with toy examples - isn't it ? The structure of the solution is the same irrespective of the complexity of data being stored within the collections. The iteration is being done at a much higher level of abstraction, independent of the types stored within the container. And as I mentioned above, flatMapTo is the secret sauce in the above solution structure that abstracts the building of the new container hiding the inner details. To get more into the innards of flatMapTo and similar higher order abstractions, including the new form of Iterable[+T], have a look at the OOPSLA paper referenced above.

Postscript: In all the snippets above, I have been explicit about all type signatures, just for the sake of easier understanding of my passionate reader. In reality, almost all of these will be inferred by Scala.

Code Commit: Joint Compilation of Scala and Java Sources

Mon, 05 Jan 2009 08:00:00 +0000

One of the features that the Groovy people like to flaunt is the joint compilation of .groovy and .java files. This is a fantastically powerful concept which (among other things) allows for circular dependencies between Java, Groovy and back again. Thus, you can have a Groovy class which extends a Java class which in turn extends another Groovy class.

All this is old news, but what you may not know is the fact that Scala is capable of the same thing. The Scala/Java joint compilation mode is new in Scala 2.7.2, but despite the fact that this release has been out for more than two months, there is still a remarkable lack of tutorials and documentation regarding its usage. Hence, this post…

Concepts

For starters, you need to know a little bit about how joint compilation works, both in Groovy and in Scala. Our motivating example will be the following stimulating snippet:

// foo.scala
class Foo
 
class Baz extends Bar

…and the Java class:

// Bar.java
public class Bar extends Foo {}

If we try to compile foo.scala before Bar.java, the Scala compiler will issue a type error complaining that class Bar does not exist. Similarly, if we attempt the to compile Bar.java first, the Java compiler will whine about the lack of a Foo class. Now, there is actually a way to resolve this particular case (by splitting foo.scala into two separate files), but it’s easy to imagine other examples where the circular dependency is impossible to linearize. For the sake of example, let’s just assume that this circular dependency is a problem and cannot be handled piece-meal.

In order for this to work, either the Scala compiler will need to know about class Bar before its compilation, or vice versa. This implies that one of the compilers will need to be able to analyze sources which target the other. Since Scala is the language in question, it only makes sense that it be the accommodating one (rather than javac).

What scalac has to do is literally parse and analyze all of the Scala sources it is given in addition to any Java sources which may also be supplied. It doesn’t need to be a full fledged Java compiler, but it does have to know enough about the Java language to be able to produce an annotated structural AST for any Java source file. Once this AST is available, circular dependencies may be handled in exactly the same way as circular dependencies internal to Scala sources (because all Scala and all Java classes are available simultaneously to the compiler).

Once the analysis phase of scalac has blessed the Scala AST, all of the Java nodes may be discarded. At this point, circular dependencies have been resolved and all type errors have been handled. Thus, there is no need to carry around useless class information. Once scalac is done, both the Foo and the Baz classes will have produced resultant Foo.class and Baz.class output files.

However, we’re still not quite done yet. Compilation has successfully completed, but if we try to run the application, we will receive a NoClassDefFoundError due to the fact that the Bar class has not actually been compiled. Remember, scalac only analyzed it for the sake of the type checker, no actual bytecode was produced. Bar may even suffer from a compile error of some sort, as long as this error is within the method definitions, scalac isn’t going to catch it.

The final step is to invoke javac against the .java source files (the same ones we passed to scalac) adding scalac’s output directory to javac’s classpath. Thus, javac will be able to find the Foo class that we just compiled so as to successfully (hopefully) compile the Bar class. If all goes well, the final result will be three separate files: Foo.class, Bar.class and Baz.class.

Usage

Although the concepts are identical, Scala’s joint compilation works slightly differently from Groovy’s from a usage standpoint. More specifically: scalac does not automatically invoke javac on the specified .java sources. This means that you can perform “joint compilation” using scalac, but without invoking javac you will only receive the compiled Scala classes, the Java classes will be ignored (except by the type checker). This design has some nice benefits, but it does mean that we usually need at least one extra command in our compilation process.

All of the following usage examples assume that you have defined the earlier example in the following hierarchy:

main

java

Bar.java

scala

foo.scala

target

classes

Command Line

# include both .scala AND .java files
scalac -d target/classes src/main/scala/*.scala src/main/java/*.java

javac -d target/classes 
      -classpath $SCALA_HOME/lib/scala-library.jar:target/classes 
       src/main/java/*.java

Ant

<target name="build">
    <scalac srcdir="src/main" destdir="target/classes">
        <include name="scala/**/*.scala"/>
        <include name="scala/**/*.java"/>
    </scalac>
 
    <javac srcdir="src/main/java" destdir="${scala.library}:target/classes" 
           classpath="target/classes"/>
</target>

Maven

One thing you gotta love about Maven: it’s fairly low on configuration for certain common tasks. Given the above directory structure and the most recent version of the maven-scala-plugin, the following command should be sufficient for joint compilation:

mvn compile

Unfortunately, there have been some problems reported with the default configuration and complex inter-dependencies between Scala and Java (and back again). I’m not a Maven…maven, so I can’t help too much, but as I understand things, this POM fragment seems to work well:

<plugin>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
 
    <executions>
        <execution>
            <id>compile</id>
            <goals>
            <goal>compile</goal>
            </goals>
            <phase>compile</phase>
        </execution>
 
        <execution>
            <id>test-compile</id>
            <goals>
            <goal>testCompile</goal>
            </goals>
            <phase>test-compile</phase>
        </execution>
 
        <execution>
            <phase>process-resources</phase>
            <goals>
            <goal>compile</goal>
            </goals>
        </execution>
    </executions>
</plugin>
 
<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.5</source>
        <target>1.5</target>
    </configuration>
</plugin>

You can find more information on the mailing-list thread.

Buildr

Joint compilation for mixed Scala / Java projects has been a long-standing request of mine in Buildr’s JIRA. However, because it’s not a high priority issue, the developers were never able to address it themselves. Of course, that doesn’t stop the rest of us from pitching in!

I had a little free time yesterday afternoon, so I decided to blow it by hacking out a quick implementation of joint Scala compilation in Buildr, based on its pre-existing support for joint compilation in Groovy projects. All of my work is available in my Buildr fork on GitHub. This also includes some other unfinished goodies, so if you want only the joint compilation, clone just the scala-joint-compilation branch.

Once you have Buildr’s full sources, cd into the directory and enter the following command:

rake setup install

You may need to gem install a few packages. Further, the exact steps required may be slightly different on different platforms. You can find more details on Buildr’s project page.

With this highly-unstable version of Buildr installed on your unsuspecting system, you should now be able to make the following addition to your buildfile (assuming the directory structure given earlier):

require 'buildr/scala'
 
# rest of the file...

Just like Buildr’s joint compilation for Groovy, you must explicitly require the language, otherwise important things will break. With this slight modification, you should be able to build your project as per normal:

buildr

This support is so bleeding-edge, I don’t even think that it’s safe to call it “pre-alpha”. If you run into any problems, feel free to shoot me an email or comment on the issue.

Conclusion

Joint compilation of Java and Scala sources is a profound addition to the Scala feature list, making it significantly easier to use Scala alongside Java in pre-existing or future projects. With this support, it is finally possible to use Scala as a truly drop-in replacement for Java without modifying the existing infrastructure beyond the CLASSPATH. Hopefully this article has served to bring slightly more exposure to this feature, as well as provide some much-needed documentation on its use.

Caoyuan Deng: A Case Study of Scalability Related "Out of memory" Crash in Erlang

Sun, 04 Jan 2009 02:59:59 +0000

We are building a platform for message switching, in Erlang. Everything looks OK on stability and features. It actually has run more than half year with zero down. We tested its performance on our 2-core CPU machine before, and got about 140 transactions/second, it's good enough.

Then, we got a 8-core CPU machine several weeks ago, and we did same performance testing on it, to see the scalability. Since Erlang is almost perfect on scalability, you can image the result, yes, about 700 transactions/second now, scaled almost linear. Until it crashed with "out of memory" when million hits processed.

It left a very big "erl_crash.dump" file there, I had to dig the issue. My first guess was, were some remote requests (access db, access remote web service etc) timeout but the process itself was not timeout yet, and cause more and more processes kept in VM?

A quick grep "=proc:" erl_crash.dump showed that the total number of processes was about 980, which was reasonable for our case.

So, which process ate so many memory? A quick grep "Stack+head" erl_crash.dump showed that there was indeed a process with 285082125 size of Stack+head there.

Following this clue, I caught this process:

=proc:0.4.0>
State: Garbing
Name: error_logger
Spawned as: proc_lib:init_p/5
Last scheduled in for: io_lib_format:pad_char/2
Spawned by: 0.1.0>
Started: Sun Apr  1 01:21:50 2012
Message queue length: 2086029
Number of heap fragments: 1234053
Heap fragment data: 281266956
Link list: [0.27.0>, 0.0.0>, {from,0.42.0>,#Ref0.0.0.88>}]
Reductions: 72745575
Stack+heap: 285082125
OldHeap: 47828850
Heap unused: 121777661
OldHeap unused: 47828850
Program counter: 0x0764c66c (io_lib_format:pad_char/2 + 4)
CP: 0x0764c1b4 (io_lib_format:collect_cseq/2 + 124)

This process was error_logger, which is from OTP/Erlang standard lib: error_logger, writing received messages to log file or tty. The typical usage is:

error_logger:info_msg("~p:~p " ++ Format, [?MODULE, ?LINE] ++ Data))

Which will format Data to a String according to the Format string, and write it to tty or log file.

The above case showed the message queue length of process "error_logger" had reached 1234053, and the Stack+heap was 285082125, about 272M size.

So the cause may be, that the message queue could not be processed in time, the messages were crowded in error_logger's process and finally caused "out of memory". The bottle-neck was that when error_logger tried to format the message to String, Erlang VM was weak on processing them, which seemed to need a lot of CPU cycles.

In my previous blog, I talked about Erlang is bad on massive text processing. Erlang processes String/Text via List, which is obvious bottle-neck in Erlang now, with Erlang is getting much and much popular and more and more Erlang applications are written.

But, why this did not happen on our 2-core CPU machine? It's an interesting scalability related problem:

"error_logger" module will registered one and only one process to receive and handle all log messages. But Erlang VM's scheduler can not distribute ONE process to use multiple CPUs' computing ability. In our 2-core machine, the whole ability is about 140 transactions/second, the one process of "error_logger" just happened to have the power to handle corresponding log messages in time. Under 8-core CPUs machine, our platform scales to handle 700 transactions/second, but there is still only one process of "error_logger", which can not use 8-core CPUs' ability at all, and finally fail on it.

Erlang treats every process fairly (although you can change the priority manually), we can do a simple/quick evaluation:

1. 2-Core machine, keeping hits at 140 trans/second:
The number of simultaneous processes will be about 200, each process shares the CPU cycles: 1/200 * 2 Core = 1%

2. 8-Core machine, keeping hits at 700 trans/second:
The number of simultaneous processes will be about 980, each process shares the CPU cycles: 1/980 * 8 Core = 0.82%

So, the CPU cycles shared by error_logger process actually not increases.

BTW, I think error_logger should cut its message queue when can not process them in time (disk IO may also be slower than receiving messages).

Rants, Raves and Ridicule: Initial thoughts on JavaFX

noreply@blogger.com (J. Suereth) — Sat, 03 Jan 2009 12:44:34 +0000

So recently I've been reading a little bit about the new "JavaFX" platform Sun has released. Specifically I've been looking into the Fx Script language. When I first heard about it, I kept thinking "OH no, *another* JVM language. I hope they don't make it too similar to java!". However, I've been pleasantly surprised by the features so far.

Based on a few tutorials, JavaFX script appears to have chosen nice features from languages like Scala and Javascript, while adding one languages feature that truly distinguishes it (and IMHO makes it a worthwhile language to learn/use for its domain). Here's a list of features I saw in the tutorials that I believe make the core of the Fx Script language:

Declarative Constructors (like javascript) -
```
{ fieldName: value }
```

Truly Useful Type inferences (with types on the right) [like scala] -
```
var radius = 5.0
```
Note: types can be specified

Three main types of constructs (similar to scala) - mutable references, immutable reference and functions
```
var mutableValue = 5.0
def someStaticValue = 2.0
function doSomething() {}
```
These made me think scala's def=> Fx's function, Scala's var=> Fx's var and Scala's val=> Fx's def

Value Binding - (this is a truly compelling feature for JavaFX)
```
var x = 0;
def y = bind x;
```
The above code means that x and y will always have the same value, even though y is immutable. This is useful for synching your model to your view. I'm excited to start toying with this. I've only shown on possible syntax, there are actually a few ways to bind variables.

Bound functions -
```
var x = 0;
def foo = bound function(...) {...};
```
The above code means that the function foo will see any changes made to "x". I.e. foo is no longer a closure in the sense that it retains the value of x during its call.

Anyway, I'm pleasantly surprised with JavaFX and the Fx Script language. I think a few good ideas from other JVM languages are finally really starting to take hold. Most importantly, I'm glad Scala isn't the only viable JVM language with really useful type-inference. I'm hoping the other statically typed languages will follow suit (This means you java!).

Ricky Clarkson: The Typeclass Pattern

noreply@blogger.com (Ricky Clarkson) — Fri, 02 Jan 2009 15:02:03 +0000

The Typeclass Pattern

This pattern applies to C# and Java, and possibly some other typed languages. It doesn't apply to C++, which does generics via templates.

A simple problem to demonstrate it with is a generic Pythagoras method that takes two numbers a and b and computes sqrt(a * a + b * b) for them. In both C# and Java this can't be written in the most straightforward way:

public T Pythagoras<T>(T a, T b)
{
    return Math.sqrt(a * a + b * b);
}

There's more than one reason, but the 'innermost' is that T doesn't have the operator *, and there's no way to restrict T so that it does.

The equivalent C++ will work, because C++ generates code rather than using constraints.

So the missing information in Pythagoras is:

* How to multiply two Ts. (let's assume T * T gives T)

* How to add two Ts. (let's assume T + T gives T)

* How to find the square root of a T.


interface Num<T>
{
    T Add(T one, T two);
    T Multiply(T one, T two);
    T Sqrt(T t);
}

Now Pythagoras can be written with an extra parameter, a Num:

public T Pythagoras<T>(T a, T b, Num<T> num)
{
    return num.Sqrt(num.Add(num.Multiply(a, a), num.Multiply(b, b)));
}

Clearly, some language support wouldn't go amiss for this, at least in the arithmetic case. What's more of a problem is that your method gets a 'surprising' extra parameter, the Num. It's one of those things that makes sense in isolation, but is really an unnecessary detail when reading the code to gain an understanding of it. Scala offers an interesting solution to that.

In Scala, you might write Pythagoras as:

def pythagoras[T](a: T, b: T)(implicit num: Num[T]) = num.sqrt(num.add(num.multiply(a, a), num.multiply(b, b)))

Then calling it would look like:

pythagoras(3, 5)

The second parameter list for pythagoras is an 'implicit' parameter list, meaning that in compilation the compiler looks for a Num[T] in scope that it can use for num. You can specify it explicitly, e.g., pythagoras(3, 5)(Num.integer), or you can just import Num.integer at some point.

As with all patterns, though, all it takes is a language to support it directly, and then the pattern disappears from user code. In Haskell, Num is a built in type class and Int, Double, etc. are types that are instances, or members, of Num. +, -, *, / are defined as methods that are part of the Num typeclass. Let's omit the sqrt part for a moment:

Prelude> let pythagorasSquared a b = a * a + b * b
Prelude> pythagorasSquared 3 4
25
Prelude> :type pythagorasSquared
pythagorasSquared :: (Num t) => t -> t -> t

I hope the first two commands I gave to the Haskell interpreter are self-explanatory. The third asks what type pythagorasSquared is. That's like asking what signature a method has.

pythagorasSquared is a function with one type parameter called t. t is an instance, or member, of the Num type class. pythagorasSquared takes two values of this type t, and returns a value of the same type. E.g., given two Doubles it will yield a Double. This usually works internally by passing around a Num implicitly, like in the Scala solution, but can work quite like the C++ code generation way, depending entirely on the compiler.

The reason that I omitted the sqrt is that in Haskell, sqrt is defined on Floating, another type class, rather than Num (giving the sqrt of an Int as an Int is perhaps not useful).

At some point, some readers will have stopped understanding the terms, but if you grasped the concept, this wasn't a waste of time!

Caoyuan Deng: The Year That Will Be

Thu, 01 Jan 2009 19:39:00 +0000

It's 2009 now, in Beijing.

1==0.999999999......

I met Erlang 2 years ago, which finally brings me to Scala. I learnt a lot from Erlang, and I entered the Scala world with Erlang atmosphere surrounding me. The FP, the Pattern Match, the Actor/Process, I found these familiar friends in Scala everywhere.

Scala has extra bonus, to me, static types and OO/FP. The domains I face are usually with a lot of business logic, or, the worlds I try to describe are not only messages, they are, models I don't think are suitable to describe in Function only.

The world itself is OO/FP mixed, like Martin's quote: Two sides of coin. It's something like the Particle/Wave in Quantum. The world is an infinite whole, but the reason of Human Being is always finite, we are using out finite reason to measure the infinite world, it's an unsolvable contradiction: Infinity vs Finite. We have to read our world in OO and, in FP, in snapshot and in continuation.

There won't be "Super Hero" in computer languages, the world is getting self-organization and harmony, so do the languages. Each language is living in an eco-system, born, growing via interacting with environment, disappear ...

The Economy

It was bad in 2008. I tried to do some computing on stock market based on my neural network. What I can say is it will be swing in the next half-year, no big drop, no big rise. The Shanghai Stock Index will swing between 1200 and 3000. At least, no big worse any more.

My Self

I need to make some big decisions in this a year.

Caoyuan Deng: Thinking in Scala vs Erlang

Tue, 30 Dec 2008 18:15:17 +0000

Keeping Erlang in mind, I've coded two months in Scala, I'm thinking something called "Scala vs Erlang", I wrote some benchmark code to prove me (the code and result may be available someday), and I'd like to do some gradually summary on it in practical aspect. These opinions may be or not be correct currently due to lacking of deep experience and understanding, but, anyway, I need to record them now and correct myself with more experiences and understanding got on both Scala and Erlang.

Part I. Syntax

List comprehension

Erlang:

Lst = [1,2,3,4],
[X + 1 || X - Lst],
lists:map(fun(X) -> X + 1 end, Lst)

Scala:

val lst = List(1,2,3,4) 
for (x - lst) yield x + 1
lst.map{x => x + 1}
lst.map{_ + 1} // or place holder

Pattern match

Erlang:

case X of
   {A, B} when is_integer(A), A > 1 -> ok;
   _ -> error
end,

{ok, [{A, B} = H|T]} = my_function(X)

Scala:

x match {
   case (a:Int, b:_) if a > 1 => OK // can match type
   case _ => ERROR
}

val ("ok", (h@(a, b)) :: t) = my_function(x)

List, Tuple, Array, Map, Binary, Bit

Erlang:

Lst = [1, 2, 3] %% List
[0 | Lst]  %% List concat
{1, 2, 3}  %% Tuple
1, 2, “abc”>>  %% Binary
%% no Array, Map syntax

Scala:

val lst = List(1, 2, 3)  // List
0 :: lst  // List concat
(1, 2, 3) // Tuple
Array(1, 2, 3) // Array
Map(“a” -> 1, “b” -> 2) // Map
// no Binary, Bit syntax

Process, Actor

Erlang:

the_actor(X) -> 
   receive 
      ok -> io:format(“~p~n”, [X]);
      I -> the_actor(X + I) %% needs to explicitly continue loop
   end.
P = spawn(mymodule, the_actor, [0])
P ! 1
P ! ok

Scala I:

class TheActor(x:Int) extends Actor { 
   def act = loop {
      react {
         case “ok” => println(x); exit // needs to explicitly exit loop
         case i:Int => x += i
      }
   }
}
val a = new TheActor(0)
a ! 1
a ! “ok”

Scala II:

val a = actor { 
   def loop(x:Int) = {
      react {
         case "ok" => println(x)
         case i:Int => loop(x + i)
      }
   }
   loop(0)
}
a ! 1
a ! "ok"

Part II. Processes vs Actors

Something I

Erlang:

Lightweight processes
You can always (almost) create a new process for each new comer
Scheduler treats all processes fairly
Share nothing between processes
Lightweight context switch between processes
IO has been carefully delegated to independent processes

Scala:

Active actor is delegated to JVM thread, actor /= thread
You can create a new actor for each new comer
But the amount of real workers (threads) is dynamically adjusted according to the processing time
The later comers may be in wait list for further processing until a spare thread is available
Share nothing or share something upon you decision
Heavy context switch between working threads
IO block is still pain unless good NIO framework (Grizzly?)

Something II

Erlang:

Try to service everyone simultaneously
But may loss service quality when the work is heavy, may time out (out of service)
Ideal when processing cost is comparable to context switching cost
Ideal for small message processing in soft real-time
Bad for massive data processing, and cpu-heavy work

Scala:

Try to service limited number of customers best first
If can not service all, the later comers will be put in waiting list and may time out (out of service)
It's difficult for soft real-time on all coming concurrent customers
Ideal when processing cost is far more than context switching cost (context switch time is in ns on modern JVM)
When will there be perfect NIO + Actor library?