The right tool for the job

James Iry recently wrote a post about type systems in which he ranked type systems according to two characteristics, safety and sophistication. We had an interesting discussion about it on IRC afterwards, in which one of the things discussed was whether these were really the right axes.

It occurred to me that we spend a lot of time talking about the major axes on which you compare programming languages, and that it would be interesting to gather some data on what these actually were instead of just plucking things that seem like they may sense out of the air. So I’ve created a little app to gather data for this: Check out The Right Tool, and I’d love it if you’d rank some languages on it.

Essentially the idea is this: I have a list of statements and a list of programming languages, and I want you to rank those languages in order of how well the statements apply to them.

At the moment I’m just data gathering: I don’t actually do anything with the data. The end goal is that I’m going to have a very high dimensional representation of each programming language in terms of what people think of them and why they use them which I will do some sort of dimension reduction to, hopefully giving rise to some interesting metrics with which to compare languages.

A couple notes:

  • This is not about which language is best. It’s about comparing the strengths and weaknesses of different languages
  • I’m not making the data available yet, but I promise I will later once I’ve gathered enough
  • The site looks like ass. I know. I’m not a web designer. Deal with it or send me a better stylesheet
  • If I’ve missed any languages out, let me know. Drop me a comment or something. Similarly if you have ideas for good statements you haven’t seen on the site
This entry was posted in Data geeking, programming on by .

39 thoughts on “The right tool for the job

  1. Harald Korneliussen

    I filled out a couple, but it’s really hard to try to rank all of these languages, some of which I have barely heard about, on all sorts of dimensions.

    Some things that would make it easier and more reliable
    1. Allow ranking two languages the same, and let everything be ranked equally in the beginning
    2. Randomize the order of the entries, or you will see high in the alphabet languages on top a lot…
    3. Most radical suggestion: exclude all but a couple of languages randomly on each question. At the most extreme, a “hot ot not”-like format where you only compare two languages(in one question) at a time. You’ll get less data, but better quality, and more people can take the time to participate at all.

  2. Orion Edwards

    This is a fascinating idea, I’d love to see what eventually comes from this :-)

    After the first question where I selected the 9 or so languages I know anything about, it was quite nice to use, but the initial act of selecting those 9 from the 40 odd candidates sucked. Drag/drop using a list which scrolls offscreen is terrible. Might I suggest a series of checkboxes for the initial “pick the languages that you know” step?

    Cheers

    1. david Post author

      Yeah, I heard that a lot and finally listened. I’ve just changed around the UI to do what you suggest: There’s an initial “pick your languages” phase and a separate UI for choosing them. It does seem to be a big improvement.

      In terms of what eventually comes of that, most of that is still nascent, but here’s a first peek at it: http://therighttool.hammerprinciple.com/languages and http://therighttool.hammerprinciple.com/statements will take you to breakdowns of statement by language and language by statement. The individual languages are fascinating.

  3. david Post author

    Bah, bloody typical. In working around a Heroku bug I inadvertently introduced a bug of my own. Ruby definitely goes down in my ranking of “software I write in this language is reliable”. ;-)

    Try again now. It should work.

  4. Rohan

    Can you add a “skip to last” button? it gets boring to rank all 64 statements.

    great effort though, looking forward to more programmers ranking this.

      1. david Post author

        I’ve added Delphi, Mozart-Oz and Coq. For reasons that make sense in my head but I can’t articulate I’ve decided against FormulaONE, Fortress and Mythryl now (there has to be some cutoff – people are already complaining about the number of languages in the system).

        I deliberately chose to omit SQL because this more about general purpose programming languages, so I don’t think it really fits.

  5. david Post author

    Did you see empty rankings there? If so, that’s by design. If not, that’s a bug and I’ll look into it.

    1. david Post author

      Hmm. That’s quite unfortunate! Thanks, I’ll look into it.

      (On the plus side, the order they’ll be in at the start is deliberately random, so hopefully this shouldn’t have introduced too much bias)

  6. Nathan E. Rasmussen

    I’ll be watching with interest.

    Any thoughts of running rank-order correlation between pairs of statements? I’d like to see which statements turn out to be synonyms/antonyms/independent based on aggregate responses.

    1. k

      A more useful thing might be to do a principal components/factor analysis on the rankings to identify major dimensions underlying the rankings, and scale the languages on those dimensions, rather than on a per-statement basis.

      1. david Post author

        I’ve tried a few variations on that: It was actually the original intent of the project. there are some more variations I could try, but the problem is that the various axes for the data are highly non orthogonal, so the principle component results are somewhat dubious.

      2. k

        I tried to respond to your response, but it didn’t let me for some reason…

        Anyway, you could rotate the loadings to a non-orthogonal solution (e.g., using promax or geomin or something), if that hasn’t been tried.

        It’s very interesting to me that the axes are best thought of as nonorthogonal (which makes sense).

  7. Anton Ertl

    Cool concept. Maybe you could provide an interface that works without Javascript
    to avoid excluding those (like me) who are to concerned (or paranoid) about security
    to enable JavaScript. I have to admit that this will probably result in a pretty cumbersome
    interface.

    1. david Post author

      Err. Pass, sorry. :-) I’ve no idea how we’d do a good pure html “rearrange these items” list.

      We might do something about this later, but it’s going to be pretty low priority.

  8. quarc

    Well, even with Javascript enabled, I cannot proceed beyond the first
    question where I tick those languages I know. After sending, I am
    presented with the same sheet (entries cleared) again.
    It is a bit boring. What strange browser configuration do you need
    for this?

    1. david Post author

      I dunno. What strange browser configuration you got? We’ve tested and verified it works fine in chrome, firefox and safari, and have tested and verified that it works with a few minor visual quirks in IE. What are you using?

  9. Dan

    Lovely project. I’d be keen to see graphical programming languages in the list so that we could get some text-vs-graphics opinions emerging. In my work (audio+music) there are visual programming languages Max/MSP and Puredata (among others), and text languages SuperCollider and Csound (among others). It’s a constant and interesting topic of discussion to compare them – text-vs-graphical is never the only difference, of course.

  10. Andrés

    Trying to add new statements gives an “Internal Server Error”.

    Another step down for the reliability ranking of Ruby! :P

  11. Anonymous

    Trying to add new statement or programming language item results in “500” error. I list it in here.

    * Programming Languages: *
    CWEB
    TeX
    LaTeX
    INTERCAL
    WEB
    METAFONT
    PostScript
    FurryScript
    TROFF
    Icoruma
    Inform
    Game Maker
    QBASIC
    GWBASIC
    Inform 7
    LLVM
    dc
    SQL
    dBase
    Csound
    Puredata
    MegaZeux
    MegaZeux Forth
    ZZT

    * Statements: *
    I DISLIKE THIS PROGRAMMING LANGUAGE
    THIS PROGRAMMING LANGUAGE IS GOOD FOR GAMES
    THIS PROGRAMMING LANGUAGE IS GOOD FOR GRAPHICS
    THIS PROGRAMMING LANGUAGE IS GOOD FOR TYPESETTING
    THIS PROGRAMMING LANGUAGE IS GOOD FOR EVERYTHING
    THIS PROGRAMMING LANGUAGE IS GOOD FOR NOTHING
    THIS PROGRAMMING LANGUAGE IS GOOD FOR AUDIO
    THIS PROGRAMMING LANGUAGE IS STUPID
    THIS QUESTION IS STUPID
    THIS PROGRAMMING LANGUAGE IS GOOD FOR DESCRIBING AND EXPLAINING PROGRAMS
    THIS PROGRAMMING LANGUAGE IS BAD FOR DESCRIBING AND EXPLAINING PROGRAMS

  12. Aaron Davies

    awesome site, it’s amazing how much valid info you already have with <150 respondents.

    any plans for visualizations? clusters of languages/features, etc., would be cool

  13. Joe

    I’m new to programming and I’ve been Googling around to get my bearings before diving deep into any one programming language. Clicking around your site was a very efficient way for me to form a general idea of the space. Thank you so much!
    I like how you’ve expanded into other domains like gin and databases. I’d like to use it to rank professors, fraternities, courses and rotation sites at my school.

Comments are closed.