Is the ArcSine transformation so asinine in the end?

Some authors documented that analyses of proportions can be performed with as few as 3 participants per group (e.g., Warton & Hui, 2011). Some also reported finiding multiple configurations with significant results using logistic regressions.

We do not think this is sensible.

Let’s generate compiled data with two groups. In the warton scenario, the first group has only successes and in the second group, 2 out of 3 participants have failure.

warton <- data.frame(
  grp = c(1,2),
  s   = c(3,1),
  n   = c(3,3)  
)

Analyzing this, the ANOPA suggests no difference…

w <- anopa( {s;n}~grp, warton)
summary( w )

##             MS  df        F        p correction    Fcorr pvalcorr
## grp   0.179169   1 2.508371 0.113243   1.166667 2.150032 0.142567
## Error 0.071429 Inf

… something quite evident from the plot:

anopaPlot(w)

Figure 1. The proportions for two groups. Error bars show difference-adjusted 95% confidence intervals.

(one tip of the confidence intervals is so off the scale that it is missing.)

Because the sample is so small, it is actually possible to enumerate all the possible results (there are 64 of them). If we allow no success or a single success in one group, and all success in the other group, there are 14 cases. 14 out of 64 is far from being exceptional, and thus, there is no significant result here, congruent with the result of the ANOPA analysis (and contradicting the results from a logistic regression).

Lets consider a more extreme result: The first group has only successes and the second, only failures (there is two such cases out of 64):

laurencelle <- data.frame(
  grp = c(1,2),
  s   = c(3,0),
  n   = c(3,3)  
)

The analyse using ANOPA says:

l <- anopa( {s;n}~grp, laurencelle)
summary(l)

##             MS  df        F        p correction    Fcorr pvalcorr
## grp   0.429938   1 6.019135 0.014152   1.166667 5.159259 0.023123
## Error 0.071429 Inf

that is, a significant result (and note that 2 out of 64 is indeed rare at the .05 threshold with a p of .031 = 2/64). The plot is congruent with this result:

anopaPlot(l)

Figure 2. The proportions for two extremely different groups. Error bars show difference-adjusted 95% confidence intervals.

The logistic regression, when applied to proportions, has very inflated type-I error rates so that this technique should be avoided. The reason is quite simple: the logit transformation is not variance-stabilizing. In fact, it exaggerate the variances across levels of population proportions.