Independence of irrelevant alternatives

From Wikipedia, the free encyclopedia - View original article

 
Jump to: navigation, search

The independence of irrelevant alternatives (IIA), also known as binary independence[1] is an axiom of decision theory and various social sciences. The term is used with different meanings in different contexts. Although they all attempt to provide a rational account of individual behavior or aggregation of individual preferences, the exact formulations differ from context to context.

In individual choice theory, IIA sometimes refers to Chernoff's condition or Sen's property α (alpha): if an alternative x chosen from a set T, and x is also an element of a subset S of T, then x must be chosen from S.[2] That is, eliminating some of the unchosen alternatives shouldn't affect the selection of x as the best option.

In social choice theory, Arrow's IIA is one of the conditions in Arrow's impossibility theorem: the social preferences between alternatives x and y depend only on the individual preferences between x and y.[3] Kenneth Arrow (1951) shows the impossibility of aggregating individual rank-order preferences ("votes") satisfying IIA and certain other reasonable conditions.

There are other requirements called IIA.

One such requirement is as follows: If A is preferred to B out of the choice set {A,B}, introducing a third option X, expanding the choice set to {A,B,X}, must not make B preferable to A. In other words, preferences for A or B should not be changed by the inclusion of X, i.e., X is irrelevant to the choice between A and B. This formulation appears in bargaining theory, theories of individual choice, and voting theory. Some theorists find it too strict an axiom; experiments by Amos Tversky, Daniel Kahneman, and others have shown that human behavior rarely adheres to this axiom.

A distinct formulation of IIA is found in social choice theory: If A is selected over B out of the choice set {A,B} by a voting rule for given voter preferences of A, B, and an unavailable third alternative X, then if only preferences for X change, the voting rule must not lead to B's being selected over A. In other words, whether A or B is selected should not be affected by a change in the vote for an unavailable X, which is irrelevant to the choice between A and B.

Voting theory[edit]

In voting systems, independence of irrelevant alternatives is often interpreted as, if one candidate (X) wins the election, and a new candidate (Y) is added to the ballot, only X or Y will win the election.

Approval voting and range voting satisfy the IIA criterion. Another cardinal system, cumulative voting, does not satisfy the criterion.

An anecdote that illustrates a violation of IIA has been attributed to Sidney Morgenbesser:

After finishing dinner, Sidney Morgenbesser decides to order dessert. The waitress tells him he has two choices: apple pie and blueberry pie. Sidney orders the apple pie. After a few minutes the waitress returns and says that they also have cherry pie at which point Morgenbesser says "In that case I'll have the blueberry pie."

All voting systems have some degree of inherent susceptibility to strategic nomination considerations. Some regard these considerations as less serious unless the voting system fails the easier-to-satisfy independence of clones criterion.

Local independence[edit]

A criterion weaker than IIA proposed by H. Peyton Young and A. Levenglick is called local independence of irrelevant alternatives (LIIA). [4] LIIA requires that both of the following conditions always hold: (1) If the option that finished in last place is deleted from all the votes, then the order of finish of the remaining options must not change. (The winner must not change.) (2) If the winning option is deleted from all the votes, the order of finish of the remaining options must not change. (The option that finished in second place must become the winner.)

An equivalent way to express LIIA is that if a subset of the options are in consecutive positions in the order of finish, then their relative order of finish must not change if all other options are deleted from the votes. For example, if all options except those in 3rd, 4th and 5th place are deleted, the option that finished 3rd must win, the 4th must finish second, and 5th must finish 3rd.

Another equivalent way to express LIIA is that if two options are consecutive in the order of finish, the one that finished higher must win if all options except those two are deleted from the votes.

LIIA is weaker than IIA because satisfaction of IIA implies satisfaction of LIIA, but not vice versa.

Apart from the voting methods that also satisfy IIA, LIIA is satisfied by very few voting methods. These include Kemeny-Young and Ranked Pairs, but not Schulze.

Criticism of IIA[edit]

IIA is too strong to be satisfied by any voting method that reduces to majority rule when there are only two alternatives. Most ranked ballot rules do so, while Approval and Range can pass IIA because they do not necessarily reduce to majority rule.[5] Any comprehensive use of strategy that makes Approval or Range reduce to majority rule when there are only two candidates will make those methods fail IIA as well. (Even if only one voter is an optimizing voter, it is possible to construct a tied or nearly-tied example to show IIA can be violated.)

Consider a scenario in which there are three candidates A, B & C, and the voters' preferences are as follows:

25% of the voters prefer A over B, and B over C. (A>B>C)
40% of the voters prefer B over C, and C over A. (B>C>A)
35% of the voters prefer C over A, and A over B. (C>A>B)

(These are preferences, not votes, and thus are independent of the voting method.)

75% prefer C over A, 65% prefer B over C, and 60% prefer A over B. Regardless of the voting method and the actual votes, there are only three cases to consider:

(It is only assumed that most voters of those majorities learn to vote the obvious optimal strategy when there are only two candidates.)

So even if IIA is desirable, requiring its satisfaction seems to allow only voting methods that are undesirable in some other way, such as treating one of the voters as a dictator. Thus the goal must be to find which voting methods are best, rather than which are perfect.

An argument can be made that IIA is itself undesirable. IIA assumes that when deciding whether A is likely to be better than B, information about voters' preferences regarding C is irrelevant and should not make a difference. However, the heuristic that leads to majority rule when there are only two options is that the larger the number of people who think one option is better than the other, the greater the likelihood that it is better, all else being equal. (See Condorcet's Jury Theorem.) A majority is more likely than the opposing minority to be right about which of the two candidates is better, all else being equal, hence the use of majority rule.

It is statistical at best; the majority is not necessarily right all the time. The same heuristic implies that the larger the majority, the more likely it is that they are right. It would seem to also imply that when there is more than one majority, larger majorities are more likely to be right than smaller majorities. Assuming this is so, the 75% who prefer C over A and the 65% who prefer B over C are more likely to be right than the 60% who prefer A over B, and since it is not possible for all three majorities to be right, the smaller majority (who prefer A over B) are more likely to be wrong, and less likely than their opposing minority to be right. Rather than being irrelevant to whether A is better than B, the additional information about the voters' preferences regarding C provide a strong hint that this is a situation where all else is not equal.

In social choice[edit]

From Kenneth Arrow,[6] each "voter" i in the society has an ordering Ri that ranks the (conceivable) objects of social choicex, y, and z in simplest case—from high to low. An aggregation rule (voting rule) in turn maps each profile or tuple (R1, ...,Rn) of voter preferences (orderings) to a social ordering R that determines the social preference (ranking) of x, y, and z.

Arrow's IIA requires that whenever a pair of alternatives is ranked the same way in two preference profiles (over the same choice set), then the aggregation rule must order these alternatives identically across the two profiles.[7] For example, suppose an aggregation rule ranks a above b at the profile given by

(i.e., the first individual prefers a first, c second, b third, d last; the second individual prefers d first, ..., and c last). Then, if it satisfies IIA, it must rank a above b at the following three profiles:

The last two forms of profiles (placing the two at the top; and placing the two at the top and bottom) are especially useful in the proofs of theorems involving IIA.

Arrow's IIA does not imply an IIA similar to those different from this at the top of this article nor conversely.[8]

Historical Remark. In the first edition of his book, Arrow misinterpreted IIA by considering the removal of a choice from the consideration set. Among the objects of choice, he distinguished those that by hypothesis are specified as feasible and infeasible. Consider two possible sets of voter orderings (R_1, ...,R_n ) and (R_1', ...,R_n') such that the ranking of X and Y for each voter i is the same for R_i and R_i'. The voting rule generates corresponding social orderings R and R'. Now suppose that X and Y are feasible but Z is infeasible (say, the candidate is not on the ballot or the social state is outside the production possibility curve). Arrow required that the voting rule that R and R' select the same (top-ranked) social choice from the feasible set (X, Y), and that this requirement holds no matter what the ranking is of infeasible Z relative to X and Y in the two sets of orderings. IIA does not allow "removing" an alternative from the available set (a candidate from the ballot), and it says nothing about what would happen in such a case: all options are assumed to be "feasible."

Examples[edit]

Borda count[edit]

In a Borda count election, 5 voters rank 5 alternatives [A, B, C, D, E].

3 voters rank [A>B>C>D>E]. 1 voter ranks [C>D>E>B>A]. 1 voter ranks [E>C>D>B>A].

Borda count (a=0, b=1): C=13, A=12, B=11, D=8, E=6. C wins.

Now, the voter who ranks [C>D>E>B>A] instead ranks [C>B>E>D>A]; and the voter who ranks [E>C>D>B>A] instead ranks [E>C>B>D>A]. They change their preferences only over the pairs [B, D], [B, E] and [D, E].

The new Borda count: B=14, C=13, A=12, E=6, D=5. B wins.

The social choice has changed the ranking of [B, A] and [B, C]. The changes in the social choice ranking are dependent on irrelevant changes in the preference profile. In particular, B now wins instead of C, even though no voter changed their preference over [B, C].

Borda count and strategic voting[edit]

Consider an election in which there are three candidates, A, B, and C, and only two voters. Each voter ranks the candidates in order of preference. The highest ranked candidate in a voter's preference is given 2 points, the second highest 1, and the lowest ranked 0; the overall ranking of a candidate is determined by the total score it gets; the highest ranked candidate wins.

We consider two profiles:

Thus, if the second voter wishes A to be elected, he had better vote ACB regardless of his actual opinion of C and B. This violates the idea of "independence of irrelevant alternatives" because the voter's comparative opinion of C and B affects whether A is elected or not. In both profiles, the rankings of A relative to B are the same for each voter, but the social rankings of A relative to B are different.

Copeland[edit]

This example shows that Copeland's method violates IIA. Assume four candidates A, B, C and D with 6 voters with the following preferences:

# of votersPreferences
1A > B > C > D
1A > C > B > D
2B > D > A > C
2C > D > A > B

The results would be tabulated as follows:

Pairwise Preferences
X
ABCD
YA[X] 2
[Y] 4
[X] 2
[Y] 4
[X] 4
[Y] 2
B[X] 4
[Y] 2
[X] 3
[Y] 3
[X] 2
[Y] 4
C[X] 4
[Y] 2
[X] 3
[Y] 3
[X] 2
[Y] 4
D[X] 2
[Y] 4
[X] 4
[Y] 2
[X] 4
[Y] 2
Pairwise election results (won-tied-lost):2-0-11-1-11-1-11-0-2

Result: A has two wins and one defeat, while no other candidate has more wins than defeats. Thus, A is elected Copeland winner.

Change of irrelevant preferences[edit]

Now, assume all voters would raise D over B and C without changing the order of A and D. The preferences of the voters would now be:

# of votersPreferences
1A > D > B > C
1A > D > C > B
2D > B > A > C
2D > C > A > B

The results would be tabulated as follows:

Pairwise Preferences
X
ABCD
YA[X] 2
[Y] 4
[X] 2
[Y] 4
[X] 4
[Y] 2
B[X] 4
[Y] 2
[X] 3
[Y] 3
[X] 6
[Y] 0
C[X] 4
[Y] 2
[X] 3
[Y] 3
[X] 6
[Y] 0
D[X] 2
[Y] 4
[X] 0
[Y] 6
[X] 0
[Y] 6
Pairwise election results (won-tied-lost):2-0-10-1-20-1-23-0-0

Result: D wins against all three opponents. Thus, D is elected Copeland winner.

Conclusion[edit]

The voters changed only their preference orders over B, C and D. As a result, the outcome order of D and A changed. A turned from winner to loser without any change of the voters' preferences regarding A. Thus, Copeland's method fails the IIA criterion.

Instant-runoff voting[edit]

In an instant-runoff election, 5 voters rank 3 alternatives [A, B, C].

2 voters rank [A>B>C]. 2 voters rank [C>B>A]. 1 voter ranks [B>A>C].

Round 1: A=2, B=1, C=2; B eliminated. Round 2: A=3, C=2; A wins.

Now, the two voters who rank [C>B>A] instead rank [B>C>A]. They change only their preferences over B and C.

Round 1: A=2, B=3, C=0; C eliminated. Round 2: A=2, B=3; B wins.

The social choice ranking of [A, B] is dependent on preferences over the irrelevant alternatives [B, C].

Kemeny–Young method[edit]

This example shows that the Kemeny–Young method violates the IIA criterion. Assume three candidates A, B and C with 7 voters and the following preferences:

# of votersPreferences
3A > B > C
2B > C > A
2C > A > B

The Kemeny–Young method arranges the pairwise comparison counts in the following tally table:

All possible pairs
of choice names
Number of votes with indicated preference
Prefer X over YEqual preferencePrefer Y over X
X = AY = B502
X = AY = C304
X = BY = C502

The ranking scores of all possible rankings are:

Preferences1. vs 2.1. vs 3.2. vs 3.Total
A > B > C53513
A > C > B35210
B > A > C25310
B > C > A52411
C > A > B42511
C > B > A2428

Result: The ranking A > B > C has the highest ranking score. Thus, A wins ahead of B and C.

Change of irrelevant preferences[edit]

Now, assume the two voters (marked bold) with preferences B > C > A would change their preferences over the pair B and C. The preferences of the voters would then be in total:

# of votersPreferences
3A > B > C
2C > B > A
2C > A > B

The Kemeny–Young method arranges the pairwise comparison counts in the following tally table:

All possible pairs
of choice names
Number of votes with indicated preference
Prefer X over YEqual preferencePrefer Y over X
X = AY = B502
X = AY = C304
X = BY = C304

The ranking scores of all possible rankings are:

Preferences1. vs 2.1. vs 3.2. vs 3.Total
A > B > C53311
A > C > B35412
B > A > C2338
B > C > A3249
C > A > B44513
C > B > A44210

Result: The ranking C > A > B has the highest ranking score. Thus, C wins ahead of A and B.

Conclusion[edit]

The two voters changed only their preferences over B and C, but this resulted in a change of the order of A and C in the result, turning A from winner to loser without any change of the voters' preferences regarding A. Thus, the Kemeny-Young method fails the IIA criterion.

Minimax[edit]

This example shows that the Minimax method violates the IIA criterion. Assume four candidates A, B and C and 13 voters with the following preferences:

# of votersPreferences
2B > A > C
4A > B > C
3B > C > A
4C > A > B

Since all preferences are strict rankings (no equals are present), all three Minimax methods (winning votes, margins and pairwise opposite) elect the same winners.

The results would be tabulated as follows:

Pairwise election results
X
ABC
YA[X] 5
[Y] 8
[X] 7
[Y] 6
B[X] 8
[Y] 5
[X] 4
[Y] 9
C[X] 6
[Y] 7
[X] 9
[Y] 4
Pairwise election results (won-tied-lost):1-0-11-0-11-0-1
worst pairwise defeat (winning votes):789
worst pairwise defeat (margins):135
worst pairwise opposition:789

Result: A has the closest biggest defeat. Thus, A is elected Minimax winner.

Change of irrelevant preferences[edit]

Now, assume the two voters (marked bold) with preferences B > A > C change the preferences over the pair A and C. The preferences of the voters would then be in total:

# of votersPreferences
4A > B > C
5B > C > A
4C > A > B

The results would be tabulated as follows:

Pairwise election results
X
ABC
YA[X] 5
[Y] 8
[X] 9
[Y] 4
B[X] 8
[Y] 5
[X] 4
[Y] 9
C[X] 4
[Y] 9
[X] 9
[Y] 4
Pairwise election results (won-tied-lost):1-0-11-0-11-0-1
worst pairwise defeat (winning votes):989
worst pairwise defeat (margins):535
worst pairwise opposition:989

Result: Now, B has the closest biggest defeat. Thus, B is elected Minimax winner.

Conclusion[edit]

So, by changing the order of A and C in the preferences of some voters, the order of A and B in the result changed. B is turned from loser to winner without any change of the voters' preferences regarding B. Thus, the Minimax method fails the IIA criterion.

Plurality voting system[edit]

In a plurality voting system 7 voters rank 3 alternatives (A, B, C).

In an election, initially only A and B run: B wins with 4 votes to A's 3, but the entry of C into the race makes A the new winner.

The relative positions of A and B are reversed by the introduction of C, an "irrelevant" alternative.

Ranked pairs[edit]

This example shows that the Ranked pairs method violates the IIA criterion. Assume three candidates A, B and C and 7 voters with the following preferences:

# of votersPreferences
3A > B > C
2B > C > A
2C > A > B

The results would be tabulated as follows:

Pairwise election results
X
ABC
YA[X] 2
[Y] 5
[X] 4
[Y] 3
B[X] 5
[Y] 2
[X] 2
[Y] 5
C[X] 3
[Y] 4
[X] 5
[Y] 2
Pairwise election results (won-tied-lost):1-0-11-0-11-0-1

The sorted list of victories would be:

PairWinner
A (5) vs. B (2)A 5
B (5) vs. C (2)B 5
A (3) vs. C (4)C 4

Result: A > B and B > C are locked in (and C > A cannot be locked in after that), so the full ranking is A > B > C. Thus, A is elected Ranked pairs winner.

Change of irrelevant preferences[edit]

Now, assume the two voters (marked bold) with preferences B > C > A change their preferences over the pair B and C. The preferences of the voters would then be in total:

# of votersPreferences
3A > B > C
2C > B > A
2C > A > B

The results would be tabulated as follows:

Pairwise election results
X
ABC
YA[X] 2
[Y] 5
[X] 4
[Y] 3
B[X] 5
[Y] 2
[X] 4
[Y] 3
C[X] 3
[Y] 4
[X] 3
[Y] 4
Pairwise election results (won-tied-lost):1-0-10-0-22-0-0

The sorted list of victories would be:

PairWinner
A (5) vs. B (2)A 5
B (3) vs. C (3)C 4
A (3) vs. C (4)C 4

Result: All three duels are locked in, so the full ranking is C > A > B. Thus, the Condorcet winner C is elected Ranked pairs winner.

Conclusion[edit]

So, by changing their preferences over B and C, the two voters changed the order of A and C in the result, turning A from winner to loser without any change of the voters' preferences regarding A. Thus, the Ranked pairs method fails the IIA criterion.

Schulze method[edit]

This example shows that the Schulze method violates the IIA criterion. Assume four candidates A, B, C and D and 12 voters with the following preferences:

# of votersPreferences
4A > B > C > D
2C > B > D > A
3C > D > A > B
2D > A > B > C
1D > B > C > A

The pairwise preferences would be tabulated as follows:

Matrix of pairwise preferences
d[*,A]d[*,B]d[*,C]d[*,D]
d[A,*]964
d[B,*]376
d[C,*]659
d[D,*]863

Now, the strongest paths have to be identified, e.g. the path D > A > B is stronger than the direct path D > B (which is nullified, since it is a tie).

Strengths of the strongest paths
d[*,A]d[*,B]d[*,C]d[*,D]
d[A,*]977
d[B,*]777
d[C,*]889
d[D,*]887

Result: The full ranking is C > D > A > B. Thus, C is elected Schulze winner and D is preferred over A.

Change of irrelevant preferences[edit]

Now, assume the two voters (marked bold) with preferences C > B > D > A change their preferences over the pair B and C. The preferences of the voters would then be in total:

# of votersPreferences
4A > B > C > D
2B > C > D > A
3C > D > A > B
2D > A > B > C
1D > B > C > A

Hence, the pairwise preferences would be tabulated as follows:

Matrix of pairwise preferences
d[*,A]d[*,B]d[*,C]d[*,D]
d[A,*]964
d[B,*]396
d[C,*]639
d[D,*]863

Now, the strongest paths have to be identified:

Strengths of the strongest paths
d[*,A]d[*,B]d[*,C]d[*,D]
d[A,*]999
d[B,*]899
d[C,*]889
d[D,*]888

Result: Now, the full ranking is A > B > C > D. Thus, A is elected Schulze winner and is preferred over D.

Conclusion[edit]

So, by changing their preferences over B and C, the two voters changed the order of A and D in the result, turning A from loser to winner without any change of the voters' preferences regarding A. Thus, the Schulze method fails the IIA criterion.

Two-round system[edit]

A probable example of the two-round system failing this criterion was the 1991 Louisiana gubernatorial election. Polls leading up to the election suggested that, had the runoff been Edwin Edwards v Buddy Roemer, Roemer would have won. However, in the election David Duke finished second and make the runoff instead of Roemer, which Edwards won by a large margin. Thus, the presence of Duke in the election changed which of the non-Duke candidates won.

In econometrics[edit]

IIA is a property of the multinomial logit and the conditional logit models in econometrics; outcomes that could theoretically violate the IIA (such as the outcome of multicandidate elections or any choice made by humans) may make multinomial logit and conditional logit invalid estimators.

IIA implies that adding another option or changing the characteristics of a third option does not affect the relative odds between the two options considered. This implication is not realistic for applications with similar options. Many examples have been constructed to illustrate this problem.[9]

Consider the Red Bus/Blue Bus example. Commuters face a decision between car and red bus. Suppose that a commuter chooses between these two options with equal probability, 0.5, so that the odds ratio equals 1. Now suppose a third mode, blue bus, is added. Assuming bus commuters do not care about the color of the bus, they are expected to choose between bus and car still with equal probability, so the probability of car is still 0.5, while the probabilities of each of the two bus types is 0.25. But IIA implies that this is not the case: for the odds ratio between car and red bus to be preserved, the new probabilities must be car 0.33; red bus 0.33; blue bus 0.33.[10] In intuitive terms, the problem with the IIA axiom is that it leads to a failure to take account of the fact that red bus and blue bus are very similar, and are "perfect substitutes".

Many modeling advances have been motivated by a desire to alleviate the concerns raised by IIA. Generalized extreme value,[11] multinomial probit (also called conditional probit) and mixed logit are models for nominal outcomes that relax IIA, but they often have assumptions of their own that may be difficult to meet or are computationally infeasible. The multinomial probit model has as a disadvantage that it makes calculation of maximum likelihood infeasible for more than five options as it involves multiple integrals. IIA can be relaxed by specifying a hierarchical model, ranking the choice alternatives. The most popular of these is the nested logit model.[12]

Generalized extreme value and multinomial probit models possess another property, the Invariant Proportion of Substitution,[13] which suggests similarly counterintuitive individual choice behavior.

Choice under uncertainty[edit]

In the expected utility theory of von Neumann and Morgenstern, four axioms together imply that individuals act in situations of risk as if they maximize the expected value of a utility function. One of the axioms is a version of the IIA axiom:

If \,L\prec M\,, then for any \,N\, and \,p\in(0,1]\,,
\,pL+(1-p)N \prec pM+(1-p)N.\,

where p is a probability and \,L\prec M\, means that M is preferred over L. This axiom says that if one outcome (or lottery ticket) L is considered to be not as good as another (M), then having a chance with probability p of receiving L rather than N is considered to be not as good as having a chance with probability p of receiving M rather than N.

In nature[edit]

Natural selection can favor animals' IIA-type choices, thought to be due to occasional availability of foodstuffs, according to a study published Jan 2014.[14]

See also[edit]

References[edit]

External links[edit]

Footnotes[edit]

  1. ^ Saari, Donald G. (2001). Decisions and elections : explaining the unexpected (1. publ. ed.). Cambridge [u.a.]: Cambridge Univ. Press. p. 39. ISBN 0-521-00404-7. 
  2. ^ Sen, 1970, page 17.
  3. ^ Arrow, 1963, page 28.
  4. ^ Equity: In Theory and Practice; by H. Peyton Young (1995)
  5. ^ Smith, W. D (2008). The "Majority criterion" and Range Voting
  6. ^ Arrow, 1951, pp. 15, 23, 27
  7. ^ More formally, an aggregation rule (social welfare function) f is pairwise independent if for any profiles p=(R_1, \ldots, R_n), p'=(R'_1, \ldots, R'_n) of preferences and for any alternatives x, y, if R_i\cap\{x,y\}^2=R'_i\cap \{x,y\}^2 for all i, then f(p)\cap\{x,y\}^2=f(p')\cap \{x,y\}^2. This is the definition of Arrow's IIA adopted in the context of Arrow's theorem in most textbooks and surveys (Austen-Smith and Banks, 1999, page 27; Campbell and Kelly, 2002, in Handbook of SCW, page 43; Feldman and Serrano, 2005, Section 13.3.5; Gaertner, 2009, page 20; Mas-Colell, Whinston, Green, 1995, page 794; Nitzan, 2010, page 40; Tayor, 2005, page 18; see also Arrow, 1963, page 28 and Sen, 1970, page 37). This formulation does not consider addition or deletion of options, since the set of options is fixed, and this is a condition involving two profiles.
  8. ^ Paramesh Ray, "Independence of Irrelevant Alternatives," Econometrica, Vol. 41, No. 5, pp. 987-991.
  9. ^ Beethoven/Debussy (Debreu 1960; Tversky 1972), Bicycle/Pony (Luce and Suppes 1965), and Red Bus/Blue Bus (McFadden 1974)
  10. ^ Wooldridge 2002, pp. 501-2
  11. ^ McFadden 1978
  12. ^ McFadden 1984
  13. ^ Steenburgh 2008
  14. ^ rsbl.royalsocietypublishing.org/content/10/1/20130935