Update: After a conversation with Paul Crowley I am somewhat rethinking my position here. I don’t think (as he does) that what I have written is obviously wrong, but I’m at least convinced it is less obviously right. Possibly more to follow later, possibly not.

So there’s a thing called the Von Neumann–Morgenstern utility theorem, which implies that if you have a bunch of states of the world and you have a transitive preference amongst lotteries amongst those states of the world (i.e. we end up in state \(A_k\) with probability \(p_k\)), plus some other assumptions about that preference, then there is a real valued utility function such that you prefer one lottery to another if its expected utility is higher.

I was thinking about this and I realised something. If you have a real valued utility function over states of the world then your decision procedure amongst lotteries should not be expected utility and the actual decision procedure you should take is as follows: Given lotteries A and B, you should prefer the lottery that you expect to give you a higher value.

Read what I said carefully.

Expect to give you a higher value is not the same relationship as give you a higher expected value.

We want to choose the A if \(P(f(A) > f(B)) > P(f(B) > f(A))\) (where \(f\) is our value function). This is not the same relationship as \(\mathbb{E}(f(A)) > \mathbb{E}f(B)\). Moreover, this relationship is non-transitive!

(Note: It may be reasonably argued that this is not true, and that where the expected values are different you should still choose the one with the higher expected value and the lower probability. However I think this doesn’t matter because we can then construct examples where the expected values are all equal and replace this with the decision procedure “Pick the highest expected value if there is one, else pick the one which maximizes this probability”)

Don’t believe me? The proof is very simple.

Pick nine outcomes \(A_1, \ldots, A_9\) such that their values are distinct (if you have fewer distinct values than this, replace some of these with lotteries to give distinct expected values, but you probably don’t have fewer distinct values than this – e.g. I suspect you assign distinct values to “N people die” for quite a wide range of N)

Now use those 9 outcomes to construct non-transitive dice. Put an outcome one each side.

Rolling these non-transitive dice  to get the outcome on their side is then a set of three lotteries over events such that the preference relationships between them based on your value function is non-transitive