The following experiment design has occurred to me. I think it would be interesting, but I’m not sure it would be quite interesting enough for me to put in the effort to create it.

The domain of experiment is as follows: Given a presumed homogenous group trying to make a decision about a shared goal, what is the best decision procedure (i.e. voting system) for combining their opinions about it into a single decision?

The setup is as follows.

We take a strategy game. I picked reversi as a nice sweet spot between complicated and simple (it also has the nice property that the number of valid moves is quite constrained). Teams of people are assigned to black and white respectively.

A turn is played as follows:

The set of valid moves is generated. If there are no valid moves or only one play proceeds in the obvious constrained fashion without user input.
If there are more than some cap of valid moves pick a random subset of them (I’m not sure this is necessary. The maximum number of valid moves in reversi shouldn’t be very large).
The team then votes in some manner on which move to take.
The move which the voting system declares to be the winner is taken.

Note that this requires that the decision procedure not allow ties. This isn’t a terrible constraint as you can just add a random tie breaker to the decision procedure.

What’s the output of this experiment?

Each game generates the following data point:

Black had N players using voting procedure X
White had M players using voting procedure Y
C won

(it also generates the complete history of the game if that’s useful or interesting, but I suspect that’s too complicated to properly analyze)

A sufficiently large sample of this data can be used to answer a number of questions. The following are particularly interesting:

1. Given the same size of group, does voting procedure X tend to beat voting procedure Y?
2. Given a fixed voting procedure X, what is P(X applied to a group of size N beats a single opponent) as a function of N?
3. Given a fixed voting procedure X, what is P(X applied to a group of size N beats X applied to a group of size M) as a function of M, N

Obviously you can’t then go on from this to say “This is the best voting procedure” as it’s applied to a very simple case, but I think the results would be interesting.