James Iry recently wrote a post about type systems in which he ranked type systems according to two characteristics, safety and sophistication. We had an interesting discussion about it on IRC afterwards, in which one of the things discussed was whether these were really the right axes.
It occurred to me that we spend a lot of time talking about the major axes on which you compare programming languages, and that it would be interesting to gather some data on what these actually were instead of just plucking things that seem like they may sense out of the air. So I’ve created a little app to gather data for this: Check out The Right Tool, and I’d love it if you’d rank some languages on it.
Essentially the idea is this: I have a list of statements and a list of programming languages, and I want you to rank those languages in order of how well the statements apply to them.
At the moment I’m just data gathering: I don’t actually do anything with the data. The end goal is that I’m going to have a very high dimensional representation of each programming language in terms of what people think of them and why they use them which I will do some sort of dimension reduction to, hopefully giving rise to some interesting metrics with which to compare languages.
A couple notes:
- This is not about which language is best. It’s about comparing the strengths and weaknesses of different languages
- I’m not making the data available yet, but I promise I will later once I’ve gathered enough
- The site looks like ass. I know. I’m not a web designer. Deal with it or send me a better stylesheet
- If I’ve missed any languages out, let me know. Drop me a comment or something. Similarly if you have ideas for good statements you haven’t seen on the site
are rankings top (most) to bottom(least) or vice versa?
I filled out a couple, but it’s really hard to try to rank all of these languages, some of which I have barely heard about, on all sorts of dimensions.
Some things that would make it easier and more reliable
1. Allow ranking two languages the same, and let everything be ranked equally in the beginning
2. Randomize the order of the entries, or you will see high in the alphabet languages on top a lot…
3. Most radical suggestion: exclude all but a couple of languages randomly on each question. At the most extreme, a “hot ot not”-like format where you only compare two languages(in one question) at a time. You’ll get less data, but better quality, and more people can take the time to participate at all.
This is a fascinating idea, I’d love to see what eventually comes from this :-)
After the first question where I selected the 9 or so languages I know anything about, it was quite nice to use, but the initial act of selecting those 9 from the 40 odd candidates sucked. Drag/drop using a list which scrolls offscreen is terrible. Might I suggest a series of checkboxes for the initial “pick the languages that you know” step?
Cheers
Yeah, I heard that a lot and finally listened. I’ve just changed around the UI to do what you suggest: There’s an initial “pick your languages” phase and a separate UI for choosing them. It does seem to be a big improvement.
In terms of what eventually comes of that, most of that is still nascent, but here’s a first peek at it: http://therighttool.hammerprinciple.com/languages and http://therighttool.hammerprinciple.com/statements will take you to breakdowns of statement by language and language by statement. The individual languages are fascinating.
When I hit “Rank!”, I get: Internal Server Error
Bah, bloody typical. In working around a Heroku bug I inadvertently introduced a bug of my own. Ruby definitely goes down in my ranking of “software I write in this language is reliable”. ;-)
Try again now. It should work.
The description talks about Right and Left hand lists¸ but I only see a single list of movable boxes. What am I doing wrong?
My fault! The description is of the old UI. I’ll fix.
Can you add a “skip to last” button? it gets boring to rank all 64 statements.
great effort though, looking forward to more programmers ranking this.
I’d like to see Delphi, Mozart-Oz, FormulaONE, Fortress, Mythryl and Coq added to the list, please.
Oh, and SQL.
I’ve added Delphi, Mozart-Oz and Coq. For reasons that make sense in my head but I can’t articulate I’ve decided against FormulaONE, Fortress and Mythryl now (there has to be some cutoff – people are already complaining about the number of languages in the system).
I deliberately chose to omit SQL because this more about general purpose programming languages, so I don’t think it really fits.
When I looked at the “see and edit your previous rankings” button, I saw rankings for the questions I’ve skipped as well.
Did you see empty rankings there? If so, that’s by design. If not, that’s a bug and I’ll look into it.
No, not empty. I think they are in order they were ranked when I clicked “Skip”
Hmm. That’s quite unfortunate! Thanks, I’ll look into it.
(On the plus side, the order they’ll be in at the start is deliberately random, so hopefully this shouldn’t have introduced too much bias)
Ok. This should be fixed now. Thanks for the bug report.
I’ll be watching with interest.
Any thoughts of running rank-order correlation between pairs of statements? I’d like to see which statements turn out to be synonyms/antonyms/independent based on aggregate responses.
Yes, it’s very much on the to do list. I just haven’t gotten around to it yet.
A more useful thing might be to do a principal components/factor analysis on the rankings to identify major dimensions underlying the rankings, and scale the languages on those dimensions, rather than on a per-statement basis.
I’ve tried a few variations on that: It was actually the original intent of the project. there are some more variations I could try, but the problem is that the various axes for the data are highly non orthogonal, so the principle component results are somewhat dubious.
I tried to respond to your response, but it didn’t let me for some reason…
Anyway, you could rotate the loadings to a non-orthogonal solution (e.g., using promax or geomin or something), if that hasn’t been tried.
It’s very interesting to me that the axes are best thought of as nonorthogonal (which makes sense).
Cool concept. Maybe you could provide an interface that works without Javascript
to avoid excluding those (like me) who are to concerned (or paranoid) about security
to enable JavaScript. I have to admit that this will probably result in a pretty cumbersome
interface.
Err. Pass, sorry. :-) I’ve no idea how we’d do a good pure html “rearrange these items” list.
We might do something about this later, but it’s going to be pretty low priority.
Comma seperated list in text box ?
eg. 3,2,1,5,4
Well, even with Javascript enabled, I cannot proceed beyond the first
question where I tick those languages I know. After sending, I am
presented with the same sheet (entries cleared) again.
It is a bit boring. What strange browser configuration do you need
for this?
I dunno. What strange browser configuration you got? We’ve tested and verified it works fine in chrome, firefox and safari, and have tested and verified that it works with a few minor visual quirks in IE. What are you using?
Oh. You have cookies disabled, right? Site won’t work without them.
Lovely project. I’d be keen to see graphical programming languages in the list so that we could get some text-vs-graphics opinions emerging. In my work (audio+music) there are visual programming languages Max/MSP and Puredata (among others), and text languages SuperCollider and Csound (among others). It’s a constant and interesting topic of discussion to compare them – text-vs-graphical is never the only difference, of course.
Trying to add new statements gives an “Internal Server Error”.
Another step down for the reliability ranking of Ruby! :P
Trying to add new statement or programming language item results in “500″ error. I list it in here.
* Programming Languages: *
CWEB
TeX
LaTeX
INTERCAL
WEB
METAFONT
PostScript
FurryScript
TROFF
Icoruma
Inform
Game Maker
QBASIC
GWBASIC
Inform 7
LLVM
dc
SQL
dBase
Csound
Puredata
MegaZeux
MegaZeux Forth
ZZT
* Statements: *
I DISLIKE THIS PROGRAMMING LANGUAGE
THIS PROGRAMMING LANGUAGE IS GOOD FOR GAMES
THIS PROGRAMMING LANGUAGE IS GOOD FOR GRAPHICS
THIS PROGRAMMING LANGUAGE IS GOOD FOR TYPESETTING
THIS PROGRAMMING LANGUAGE IS GOOD FOR EVERYTHING
THIS PROGRAMMING LANGUAGE IS GOOD FOR NOTHING
THIS PROGRAMMING LANGUAGE IS GOOD FOR AUDIO
THIS PROGRAMMING LANGUAGE IS STUPID
THIS QUESTION IS STUPID
THIS PROGRAMMING LANGUAGE IS GOOD FOR DESCRIBING AND EXPLAINING PROGRAMS
THIS PROGRAMMING LANGUAGE IS BAD FOR DESCRIBING AND EXPLAINING PROGRAMS
I forgot some programming languages:
Lisp
C–
BASIC
bc
A+
Cyclone
awesome site, it’s amazing how much valid info you already have with <150 respondents.
any plans for visualizations? clusters of languages/features, etc., would be cool
Here is a big one you forgot. Actionscript ( 3 ) You have Haxe… but they are very different.
I second the nomination of AS3 (Actionscript 3).