2.1.1 Paired Student’s t-test

The paired sample t-test, sometimes called the dependent sample t-test, is a statistical procedure used to determine whether the mean difference between two sets ( $\mu_D$ ) of observations is zero. In a paired sample t-test, each subject or entity is measured twice, resulting in pairs of observations.

As a parametric procedure, the paired sample t-test makes several assumptions. Although t-tests are quite robust, it is good practice to evaluate the degree of deviation from these assumptions in order to assess the quality of the results. These assumptions include that the dependent variable should be approximately normally distributed. To test this assumption, a variety of methods are available, including graphic inspections (histogram and qqplot) or hypothesis tests such as Kolmogorov–Smirnov and Shapiro–Wilk.

In our work, to compare the accuracies of two classifiers we will use a two-tailed test and the hypotheses are defined as follows:

(6) \begin{eqnarray} \left\{\begin{array}{cc}H_0: & \mu_D = 0\\H_1: & \mu_D \neq 0\end{array}\right .\end{eqnarray}

The test statistic is calculated as:

(7) \begin{eqnarray} T=\frac{\overline{D}-\mu_D}{\sqrt{\frac{s_d^2}{n}}},\end{eqnarray}

where n is the sample size and $s_d^2$ is the variance of the sample differences.

Under null hypothesis, the test statistic

(8) \begin{equation}T \sim t_{(n-1)},\end{equation}

and the p-value will be obtained by

(9) \begin{eqnarray}\text{{p}-value} = P(T \geq |t||H_0) + P(T \leq -|t| |H_0) = 2 \times P(T \geq |t||H_0). \end{eqnarray}

2.1.2 Wilcoxon signed rank test

The Wilcoxon signed rank test is a non-parametric hypothesis test used to compare two related samples. It can be used as an alternative to the paired Student’s t-test when the population is not normally distributed.

The hypotheses can be written in terms of the median ( $\delta_D$ ) as follows:

(10) \begin{eqnarray} \left\{\begin{array}{cc}H_0: & \delta_D = 0\\H_1: & \delta_D \neq 0\end{array}\right .\end{eqnarray}

To compute the test statistic W, first we assign a rank to each absolute value of the difference between the classifiers’ accuracies, where the smallest difference gets rank 1, the second smallest gets rank 2, etc. Each rank is assigned the sign of its corresponding difference, to identify positive and negative ranks. Then, the test statistic for the Wilcoxon signed rank test is given by

\begin{equation*}W = \min\{W^+,W^-\},\end{equation*}

where $W^+$ is the sum of the positive ranks and $W^-$ is the sum of the negative ranks. If the null hypothesis $H_0$ is true, each rank has the same probability of being a positive or negative one, that is, $W^+$ and $W^-$ would be similar. Hence, small values of W would lead to the rejection of $H_0$ . The decision can also be made based on the p-value:

(11) \begin{equation}p-value=P(W\leq w \mid H_0).\end{equation}

The p-value in Equation (11) could be obtained from tables for small samples and W could be approximated by a normal distribution for large samples.

3.1.1 Conclusion drawn from the p-value only

Let X and Y be the samples of accuracies obtained by 1-NN and 3-NN, respectively, and let ( $x_i$ , $y_i$ ), $1\leq i\leq 10$ , be the 10 pairs of accuracies obtained by 1-NN and 3-NN in the 10-fold cross-validation. In addition, let $d_i=x_i-y_i$ , $1\leq i\leq 10$ , be the corresponding differences between the paired accuracies. First, we need to verify the assumption of normality of the distributions of X and Y required for applying the matched pairs t-test. This assumption is checked by using the Kolmogorov–Smirnov test for each distribution, with the following null ( $H_0$ ) and alternative ( $H_1$ ) hypothesis for each classifier.

(17) \begin{eqnarray} \left\{\begin{array}{cl}H_0: & {The accuracies have normal distribution}\\H_1: & {The accuracies do not have normal distribution}\end{array}\right .\end{eqnarray}

Clearly, accuracy values are limited to be between 0 and 1, so they are not strictly normally distributed. Despite this, the normality test is performed, because the distribution of X or Y can be close to the normal one (it can be symmetric with the same mean and median), in which case the use of the t-test would be acceptable.

The results of the Kolmogorov–Smirnov test at the 5% significance level are that the null hypothesis (representing a normal distribution) cannot be rejected, with p-values of $0.68$ and $0.83$ for the accuracy distributions of 1-NN and 3-NN, respectively. Hence, we proceed by applying the t-test (also at the 5% significance level) for the matched pairs with unknown variances.

We use a two-sided t-test, since our goal is to detect whether or not a classifier is more accurate than another, regardless of which classifier is better. Hence, the null hypothesis is that the population mean of the 1-NN’s accuracies is equal to the population mean of the 3-NN’s accuracies. The alternative hypothesis is that these two population means are different. That is:

(18) \begin{eqnarray} \left\{\begin{array}{cc}H_0: & \mu_D = 0, \\H_1: & \mu_D \neq 0,\end{array}\right .\end{eqnarray}

where $\mu_D$ is the difference between the population means of the 1-NN’s and 3-NN’s accuracies.

The t-test statistic is calculated as defined in Equation (7), so in our case this statistic is calculated as follows, where $\overline{d}=-2.08$ is the observed difference in the sample accuracies:

(19) \begin{equation}t=\frac{\overline{d}-0}{\frac{s_d}{\sqrt{n}}} = \frac{-2.08}{\frac{2.95}{\sqrt{10}}} = -2.24,\end{equation}

since

(20) \begin{equation}s_d=\sqrt{\frac{\sum_{i=1}^n(d_{i}-\overline{d})^2}{n-1}} = \sqrt{\frac{78.12}{10-1}} = 2.95.\end{equation}

Using Equation (9) to calculate the p-value, we obtain $2 \times P(t \geq |t||H_0) = 2 \times 0.0261=0.0522$ . Since the p-value is the probability of obtaining values more extreme than the observed one when $H_0$ is true, we need to add up the probabilities of values more extreme than the calculated one for a t-distribution with nine degrees of freedom.

Hence, since the p-value $\,=0.052$ is greater than the significance level $\alpha=0.05$ , the result of the matched-pairs t-test does not provide evidence for rejecting the null hypothesis that the population means of the 1-NN’s and 3-NN’s accuracies are equal.

3.1.2 Interpreting effect sizes

The value of an effect size measure can complement the p-value in a significance analysis. In our above case study, the Cohen measure of effect size, defined in Section 2.3, is calculated as $d'cohen=\frac{|t|}{\sqrt{n}}=\frac{2.24}{\sqrt{10}}=0.71$ , where t is the t-test statistic and n is the number of matched pairs (sample size).

Table 1 shows that $d'cohen=0.71$ represents an effect of medium size. That is, although the t-test’s result does not indicate a significant difference between the mean accuracies of 1-NN and 3-NN, the difference between these means has medium size, indicating that the magnitude of the difference may be relevant. Hence, the p-value and the d’cohen value indicate different results. In this case, the calculation of the power of the test, discussed in the next subsection, provides additional information required for making a more principled decision.

3.1.3 Calculating the power of the test

As discussed in Section 2.2, the power of the test represents the probability of correctly rejecting $H_0$ , that is, the probability of concluding that a pair of classifiers have different mean accuracies, when they actually do. The power of the test is actually a function because, when $H_0: \mu_D = 0$ is false, we do not know the actual value of $\mu_D$ , we only know it is different from zero and so the power of the test is calculated for all possible values of $\mu_D$ .

In Figure 1, the curve for $n=10$ represents the test power function for the possible values of the difference between the population means of the accuracies of the pair of classifiers being compared, that is, for all possible values of $\mu_D$ . In addition to the curve for $n=10$ (sample size in our case study), the figure also shows the curves representing the test power functions for two hypothetical larger sample sizes (numbers of paired accuracies), $n=25$ and $n=50$ . In the figure, the point marked in the curve for $n=10$ refers to the power of the test if the actual difference between the population mean accuracies was $-2.08$ (the observed sample difference). In this case, the power of the test (calculated via Equation (12)) is $0.49$ , which is the value for the point marked in the figure.

Figure 1

Curves for the power of the test with several n (sample size) values

That is, if the actual difference of population mean accuracies between the two classifiers is $-2.08$ , the probability of the test indicating that the classifiers’ mean accuracies are different (correctly rejecting $H_0: \mu_D = 0$ ) is only 49%.

In addition, Figure 1 shows that, for a fixed $\mu_D$ value, the power of the test increases with the sample size (n). For instance, for $\mu_D=-2.08$ , when n = 10, 25, 50, the power of the test is 49%, 92% and 98%, respectively.

As discussed earlier, the t-test’s result (p-value = $0.052$ ) did not allow us to reject the null hypothesis, so we could not conclude the population means of the 1-NN’s and 3-NN’s accuracies are different at the significance level of 5%. However, the calculated medium effect size ( $d'cohen=0.71$ ) indicates that the magnitude of that difference can be important. That is, the two measures can lead to different conclusions about the difference in the classifiers’ accuracies.

In the above case study, the power of the test (only 49%) allow us to understand why the p-value and the d’cohen measures led to different results. That is, the t-test was used despite having relatively little power for the available sample size (10 accuracy pairs). Hence, this case study shows not only the importance of considering the three measures as a whole but also the risk of making a wrong decision based on the p-value only. It also shows the drawback of using a hypothesis test without knowing its power for the available samples.

In such cases, when the effect size is potentially relevant, the researcher should not abandon the study because the result of a test with little power did not indicate statistical significance, which could lead to missing an important result.

3.2.1 Conclusion based on the p-value

Let X and Y be the samples of accuracies obtained by 1-NN and 3-NN, respectively. Consider $(x_1,y_1)$ , $(x_2,y_2)$ , …, $(x_{n},y_{n})$ , where $n=30$ , the 30 observed accuracy pairs, with corresponding differences $d_i=x_i-y_i$ , where $1\leq i\leq n$ . We will apply the two-sided, matched-pairs Wilcoxon test with the significance level of 5%, whose hypotheses are shown in Equation (21), where $\delta_D$ is the median of the differences between the population mean accuracies of the 1-NN and 3-NN classifiers in the Wholesale3 dataset.

(21) \begin{eqnarray} \left\{\begin{array}{cc}H_0: & \delta_D = 0\\H_1: & \delta_D \neq 0\end{array}\right .\end{eqnarray}

Hence, the null hypothesis is that there is no difference between the medians of the population accuracies of 1-NN and 3-NN, whilst the alternative hypothesis is that there is a difference between those medians.

First, we calculate the ranks for the absolute values of the observed accuracy differences, sorted in increasing order. The value of n is recalculated, since 13 accuracy pairs are ignored due to having a null difference, so that $n=17$ . The W statistic is the minimum between the sum of positive ranks and the sum of negative ranks, which in our case is the former sum, viz.: $W=3.5+11+15=29.5$ .

In this case, the p-value obtained with the Wilcoxon test is $0.025$ , which is smaller than the significance level $\alpha=0.05$ , so there is evidence to reject the null hypothesis. Hence, we can claim, at this significance level, that the medians of the population accuracies of 1-NN and 3-NN are significantly different.

3.2.2 Interpreting effect sizes

As noted earlier, the value of an effect size measure can complement a p-value in a significance analysis. In our case study, the effect size measure r, defined in Section 2.3, is calculated as: $r=\frac{|z|}{\sqrt{N}}=\frac{2.24}{\sqrt{60}}=0.28$ , where z is the Wilcoxon test statistic approximated by the normal distribution and N is the sum of the sample sizes, that is, twice the number of matched accuracy pairs ( $N=2\times n$ ).

As shown in Table 2, $r=0.28$ is a small effect size. Hence, the hypothesis test’s result suggests that the difference between the medians of the population accuracies of 1-NN and 3-NN is significant, but the small effect size suggests that the magnitude of this difference may not be relevant for the researcher. In this case, the p-value and the effect size measure suggest different conclusions. Again, the calculation of the power of the test, discussed in the next subsection, provides additional information required for making a more principled decision.

3.2.3 Calculating the power of the test

To use the Wilcoxon test, it is not necessary to make any assumption about the distribution of the population, since it is a non-parametric test. However, the calculation of the power of the test requires that the distribution of the accuracy differences be known, which will be obtained via simulation.

We simulated 1000 pairs of samples, each with sample size 30, randomly drawn from a normal distribution with the mean and standard deviation of the corresponding samples X and Y. For each sample pair, we applied a Wilcoxon test and then measured the proportion of significant results (where p-value < 0.05), which was $43.3\%$ (433 out of 1000). Hence, the power of the test (for $n=30$ ) is $43.3\%$ . That is, when 1-NN and 3-NN have a median population accuracy difference of $-3.36$ (observed sample difference), the probability of the test indicating that the classifiers have different accuracies is about 43%.

As noted earlier, the Wilcoxon test provided evidence to reject the null hypothesis (p-value = 0.025), indicating a significant difference between the classifiers’ accuracies. However, the $r=0.28$ measure indicates that the effect size for that accuracy difference is small and can be a non-relevant result. Hence, the p-value and the effect size measure can lead to different conclusions.

Again, the calculation of the power of the test allows us to understand why the two measures led to different conclusions. The relatively small power of the test, 43%, when the population accuracy difference is equal to the samples accuracy difference, shows that the test has a relatively small probability of indicating that the classifiers have different accuracies when that different actually exists. That is, in this case, the Wilcoxon test was used despite its small power for this sample size. Therefore, like in the case study with the t-test, this case study shows not only the importance of calculating the three measures but also the risk of making a wrong decision based on the p-value only, and the misleading conclusion that can be drawn if the power of the test is unknown for the current sample.

4.1.1 Analysis of the behavior of p-values

One of the criticisms made to the p-value is that it is influenced by many features of the study, including the sample size (Snyder & Lawson Reference Snyder and Lawson1993). Hence, we now study the sensitivity of p-values to sample size variations when comparing classifiers.

The results from the comparison of the accuracies obtained by SVM and 1-NN in the dataset ‘Contraceptive Method Choice’, for example, show this sensitivity. In this scenario, varying the sample size (number of cross-validation folds) from 10 to 20 and 30, the obtained p-values were $0.10$ , $0.004$ and $0.002$ , respectively. That is, with 10 folds, we could not conclude that the classifiers had different accuracies, but as the sample size was increased to 20 and 30, we can conclude that the difference of accuracies between the two classifiers was statistically significant (at the 5% level).

Table 4 shows that, as expected, this sensitivity (p-value reduction with sample size increase) was quite frequent in the analyzed cases. For each pair of classifiers, this table presents the number of tests executed, according to the requirements established in Section 4.1, and the percentage of these tests where the p-value fell as the sample size increased (from 10 to 20, from 10 to 30 and from 20 to 30). This percentage represents the sensitivity in question. For example, $92.86\%$ of the 28 comparisons conducted using SVM and 1-NN presented a reduction of the p-value with the increase of the sample size from 10 to 20. It is worth remembering that only the comparisons that satisfied the requirements presented in Section 4.1 were considered in this analysis.

Table 4

Percentage of applied t-tests in which the p-value fell with the increase in sample size (from 10 to 20, from 10 to 30 and from 20 to 30)

Table 4 shows that, for some pairs of classifiers, the p-value decreased with a frequency higher than for other pairs. However, on average (see the last row of the table), the p-value does tend to decrease as the sample size goes up. This reduction is not a strict rule; there are many exceptions. Probably because, in the larger samples, the new estimates of statistics such as the sample mean and the variance can be different from the estimates of the smaller sample, which can influence the new observed p-values.

Now, we analyze the percentage of cases where the decrease of the p-value—as a consequence of the sample size increase—changed the conclusion drawn from the hypothesis test, that is, when the result changed from non-significant (p-value $\geq$ 0.05) to significant (p-value < 0.05). When the sample size increased from 10 to 20, from 10 to 30 and from 20 to 30, the percentages of tests with this conclusion change were $8.7\%$ , $11.96\%$ and $10.81\%$ , respectively. These values are reported in Table 5 and show that a hypothesis test’s sensitivity to sample sizes can affect a substantial number of research results.

Table 5

Percentages of results where the conclusion changed as the sample size increased, when using the t-test

These results confirm that the sensitivity of the p-value to the sample size is indeed a reason for criticism. Another criticism is that the p-value does not measure the importance (strength) of the result; it only indicates if the result has statistical significance. According to Berben et al. (Reference Berben, Sereika and Engberg2012), the use of the p-value as sole measure can lead researchers to confuse statistical significance with practical or scientific relevance. A statistically significance result does not ensure that the difference is large or relevant in practice, and this should be clear when interpreting the result of the test.

One way to complement the analysis is to use a measure of the effect size, since making decisions based on isolated p-values can lead to misleading conclusions, as will be shown in the next subsection.

4.1.2 Interpreting effect sizes

We now add to the study the effect size measure d’cohen, which is calculated as shown in Equation (15), in Section 2. This measure will be used to evaluate the relevance of the difference between the mean accuracies of the classifiers.

A major reason to measure the effect size along with a test’s p-value is that, unlike the p-value, the effect size is not very sensitive to the sample size. This issue has also been discussed in other areas, for example, in Fern and Monroe (Reference Fern and Monroe1996), Fritz et al. (Reference Fritz, Morris and Richler2012) and Sullivan and Feinn (Reference Sullivan and Feinn2012).

While the p-value indicates whether there is evidence that the difference is really observed in the populations, the effect size indicates the relevance of this difference in the population. Thus, the p-value and the effect size evaluate different aspects of a result and, as noted earlier, may point to opposite paths: p-value indicating statistical significance and effect size small or insignificant, or p-value not indicating statistical significance and effect size medium or large. These are the so-called special cases, discussed in Section 3.

Two analyses are conducted next: the first one studies the behavior of the effect size with an increase in sample size, to compare it with the behavior of the p-value for the same increase, and the second one studies the frequency of the so-called special cases (where there is a disagreement between the p-value and the effect size) in the observed results.

In the first analysis, Table 5 shows the percentages of tests where a change in the p-value and effect size results led to a change in the conclusion, as the sample size increased. The first row shows the percentages of tests whose results were not significant but, with the increase of the sample size, became significant. The second row shows the percentages of tests that indicated small or insignificant effect size and whose results changed to medium or large as the sample size increased, from 10 to 20, from 10 to 30 and from 20 to 30. For example, with the increase of the sample size from 10 to 20, 8.70% of the non-significant cases became significant and 4.29% of the cases with small or insignificant effect size became cases with medium or large effect size.

As shown in Table 5, the percentage of tests where the effect size changed from small or insignificant to medium or large was considerably smaller than the percentage of tests where the p-value result changed from non-significant to significant, for the three cases of an increase in sample size. Note that, despite not being directly sensitive to the sample size, the effect size values have increased or decreased with the increases in sample size. This is due to the estimates (mean and variance) for larger samples being different from the ones for smaller samples—and not due to the use of larger sample sizes.

To illustrate the difference between the behavior of the p-value and the effect size when increasing the sample size, a specific case is presented: the previous comparison between the accuracies obtained by the SVM and 1-NN classifiers using the dataset ‘Contraceptive Method Choice’. With sample size 10, the p-value was higher than 0.05 and, with sizes 20 and 30, it was smaller than 0.05. However, the effect size did not change with the increase of the sample size. The effect size was $0.59$ , $0.72$ and $0.60$ for sample sizes 10, 20 and 30, respectively—all of these rated as a medium effect size.

The second and main analysis focuses on the special cases (where there is disagreement between the p-value and the effect size). In this study, cases were found where the difference between the means of the population accuracies is statistically significant, although the effect size of this difference is small. Cases were also found where there is no statistical significance for this difference and the effect size is medium. And, of course, there were also cases where these two measures agree: cases with statistical significance and a medium or large effect size, and cases with no statistical significance and with a small or insignificant effect size.

Still regarding the comparison of SVM and 1-NN using the dataset ‘Contraceptive Method Choice’, for the sample sizes 20 and 30, the conclusions for the p-value and effect size agree: the difference between the population accuracy means is statistically significant and the effect size is medium. For the sample size 10, the difference between the population accuracy means is not significant, but the effect size is medium—characterizing a special case.

Figure 2 shows the behavior of the p-value and the effect size for each sample size. The shadowed areas set the boundaries for the special cases and each point in the curves represents a performed test. All of the 1509 test runs are depicted. Each curve represents a different sample size. For example, each point in the curve made of circles represents a test conducted with sample size 10 and the points in the shadowed area represent cases with no statistical significance and medium effect size.

Figure 2

Representation of the p-values for t-tests applied to samples sized 10, 20 and 30, and the respective values of effect size—the cases in the shadowed areas represent the special cases

As shown in Figure 2, in most cases the p-value and the effect size agree: the p-value is significant (p-value < 0.05) and the effect size is medium or large (d’cohen $\geq$ 0.5), or the p-value is not significant (p-value $\geq$ 0.05) and the effect size is small or insignificant (d’cohen < 0.5). However, for the three sample sizes, special cases were also found where these measures do not agree. The percentage of special cases, and consequently the percentage of agreement cases, is shown in Table 6 (discussed later).

Table 6

Percentage of datasets with no statistical significance in the t-test and with a medium effect size, and percentage of datasets with statistical significance and small effect size, for each pair of classifiers, per sample size

These special cases are in the shadowed area of Figure 2. Note that the tests with sample size 10 had a specific type of disagreement, whilst tests with sizes 20 and 30 had another kind of disagreement. With sample size 10, in some cases the t-test’s p-value did not allow us to conclude that the classifiers have different accuracies, but the d’cohen value suggested that the effect size is medium. With sample sizes 20 and 30, in some cases the t-test’s result gave us evidence to reject the null hypothesis (p-value < 0.05), concluding that the classifiers have different accuracies, but the effect size was small.

Table 6 shows, for each pair of classifiers and each sample size, the percentage of datasets (out of the 50 datasets used in our experiments) where each of the following types of special cases occurred: (i) cases with no statistical significance in the t-test and a medium effect size, and (ii) cases with statistical significance in the t-test and a small effect size.

The results clearly show that cases with no statistical significance (p-value $\geq$ 0.05) and a medium effect size (indicating a potentially relevant difference in accuracies) were found only when the cross-validation used 10 folds. The percentage of datasets with this type of special case was over 10% for 6 of the 15 classifier pairs, and it was on average (over all classifier pairs) $8.8\%$ .

In Table 6, we only find cases with a statistical significance in the t-test and a small effect size for tests with sample sizes 20 and 30. Besides, this type of special case was observed in all pairs of classifiers for sample size 30. As for the sample size 20, this type of special case was found for only four pairs of classifiers and with low frequencies. On average, the percentages of datasets where the special case of statistical significance and a small effect size was found, for sample sizes 20 and 30, were $1.4\%$ and $9.6\%$ , respectively.

It should be noted that the special cases do not necessarily indicate an error, since the p-value and the effect size measure different properties. However, this disagreement is evidence that the data deserve more attention. Had we used only the p-value, this might have led to a wrong conclusion. The power of the test is another measure that should be calculated in any study that relies on statistical significance testing, to evaluate the adequacy of the test in the context it is being applied.

4.1.3 Evaluating the power of the test

Now, we add to the analysis the power of the Student’s t-test. As shown in Equation (12) (Section 2), the power of the test is the probability of rejecting the null hypothesis for a given $\mu_D$ . In this work, the power of the test is the probability of the test to indicate that classifiers A and B have different accuracies, given the actual difference (estimated by the observed samples difference).

The application of a hypotheses test should always be accompanied by the calculation of the power of the test, to measure its efficiency in the available sample.

Hence, the behavior of the power of the test will be evaluated along with the p-value and the effect size. To that end, we will consider cases where the p-value and the effect size agree, and also the special cases.

At first, we will verify the power of the t-test for the scenario discussed earlier (comparing SVM’s and 1-NN’s accuracies in the ‘Contraceptive Method Choice’ dataset). As reported in Table 7, the p-values were $0.10$ , $0.004$ and $0.002$ , and the d’cohen values were $0.59$ , $0.72$ and $0.60$ for sample sizes 10, 20 and 30, respectively. For sample size 10, the difference between the classifiers’ accuracies was not statistically significant, but the effect size was medium. The power of the test in this case was $0.34$ , a low power (the recommended value is over $0.8$ ), suggesting that the test was not applied in a ideal scenario, that is, the sample is small. For sample sizes 20 and 30, the t-test indicated a significant difference between the classifiers’ accuracies, whilst the d’cohen value suggests that this difference has a medium effect size. To reinforce these conclusions, the power of the test was $0.87$ and $0.89$ for sample sizes 20 and 30, respectively, that is, the test has a high probability of rejecting the null hypothesis when it is actually false.

Table 7

Power of the t-test, p-value and effect size, when comparing SVM’s and 1-NN’s accuracies in the ‘Contraceptive Method Choice’ dataset

Next, in order to facilitate the analysis, we study how the power of the test behaves in each of the following four groups of results:

1. • Group 1: With statistical significance and medium or large effect size

2. • Group 2: Without statistical significance and small or insignificant effect size

3. • Group 3: Without statistical significance and medium or large effect size

4. • Group 4: With statistical significance and small or insignificant effect size

In Groups 1 and 2, the t-test’s p-value and the effect size are in agreement, whilst in Groups 3 and 4, these two measures point to opposite conclusions, that is, they are special cases.

Figure 3 shows the boxplots for the values of the power of the test for these groups, for each sample size (10, 20 and 30). The boxplot is a graphical representation used to evaluate the empirical distribution of a set of values, where the centre box is made by the first, second (median) and third quartiles, $Q_1$ , $Q_2$ and $Q_3$ , respectively. The lower and upper bars extend, respectively, from the lower quartile until the smallest value not exceeding the lower limit $(Q_{1}-1.5(Q_{3}-Q_{1}))$ and from the upper quartile to the highest value not exceeding the upper limit $(Q_{3}+1.5(Q_{3}-Q_{1}))$ . The points outside these boundaries are considered outliers, represented by circles.

Figure 3

Boxplots for the power of the t-test, per group and sample size

Figure 3 shows that, in Group 1, with statistical significance and a medium or large effect size, in general the power of the t-test was high, with only a few outliers scoring below 80% for the three sample sizes. In this scenario, the three measures reinforce each other: the t-test’s high power gives credibility to its result (a significant difference between the classifiers’ accuracies), and the effect size indicates that this difference is large or relevant.

In Group 2, opposite results were found. The t-test did not reject the null hypothesis and the effect size is small, agreeing with the test’s result. However, the power of the test is small, indicating a low probability of the test rejecting the null hypothesis when it is really false.

Figure 3 also shows that, for the Group 3, containing only cases with sample size 10, the power of the test was low, suggesting that the sample size is too small for applying the t-test. Regarding Group 4, containing only cases with sample sizes 20 and 30, the power of the test (between 50% and 60% for sample size 20 and between 50% and 70% for size 30, approximately) indicates that the probability of the test rejecting the null hypothesis when it is false is only around 60% for these cases. As the effect size is low, we can make an analogy with the example of the aspirin study given in the Introduction: the difference is statistically significant but may not have practical significance.

Note that all the cases with a high power of the test ( $>0.8$ ) belong to Group 1, that is, cases with statistical significance in the t-test and a medium or large effect size, as shown in Figure 3. To better understand this result, recall that, as shown in Subsection 2.2, the parameter $\mu_D$ (the difference between population means) is one of the factors that influence the power of the test function, and we used the estimate $\mu_d$ (difference between sample means) to calculate this function.

The highest values of sample differences belong to Group 1. For these higher values, it is more likely that the test will obtain a statistical significance and have a medium or higher effect size, as well as a high power of the test. Indeed, these types of results were observed for the three measures. On the other hand, for small sample differences, it is less likely that these three measures could produce these results, although that is not entirely impossible.

In Group 3, even with the small sample differences, the effect size was medium or large. This result can be explained by the small standard deviation for the differences between the sample accuracies. As for Group 4, even with the small sample differences, the t-test showed statistical significance, indicating that there was a difference between the classifiers’ accuracies.

4.2.1 Analysis of the behavior of the p-value

In this subsection, we analyze the impact on the p-values obtained with the Wilcoxon test for different sample sizes when evaluating classifiers, as done in Subsection 4.1.1 for the p-values obtained with the t-test.

The same scenario used in Section 4.1 (to discuss the results for the t-test) will be used here for discussing the results for the Wilcoxon test: the comparison of the accuracies obtained by SVM and 1-NN in the ‘Contraceptive Method Choice’ dataset. We first analyze the sensitivity of the p-value to the sample size. With sample sizes 10, 20 and 30 (cross-validation with 10, 20 and 30 folds) the p-value was $0.08$ , $0.003$ and $0.004$ , respectively. That is, the difference between the classifiers’ accuracies was not significant with sample size 10, but the difference was significant with sample sizes 20 and 30.

Table 8 shows that, as expected, this p-value reduction with sample size increase was quite frequent across the conducted tests. For each pair of classifiers, the table shows the number of tests executed, according to the requirements established in Section 4.2, and the percentage of the conducted tests in which the p-value fell with the increase of the sample size, from 10 to 20, from 10 to 30 and from 20 to 30. For example, in about 82% of the 50 tests executed in the comparison between SVM and 1-NN, the p-value was reduced when the sample size increased from 10 to 20. Opposite from the analysis with the Student’s t-test conducted in Section 4.1.1, in the current analysis, almost all datasets, that is, almost all comparisons, were considered, since the Wilcoxon test is non-parametric and the only requirement is that the samples do not have a variance equal to zero.

Table 8

Percentage of applied Wilcoxon tests in which the p-value fell with the increase in sample size (from 10 to 20, from 10 to 30 and from 20 to 30)

Table 8 shows that, for some pairs of classifiers, the p-value decreased more often than for other pairs. However, on average (see the last row of the table), the p-value does tend to decrease as the sample size goes up. However, this reduction is not a strict rule; there are many exceptions. As discussed earlier, this is likely because, in larger samples, the estimates of the sample mean and variance tend to be different from the estimates for smaller samples.

As in the earlier t-test analyses, the results reported here show that the p-value obtained with the Wilcoxon test is indeed sensitive to the sample size.

Next, we evaluate in how many cases the reduction of the p-value changed the Wilcoxon test’s conclusion from non-significant to significant. The percentage of cases with such conclusion change, as the sample size increased from 10 to 20, from 10 to 30 and from 20 to 30, was $14.52\%$ , $18.15\%$ and $14.69\%$ , respectively. These values are reported in Table 9. Although they seem low if compared to the observed percentages of p-value reductions, they show that such sensitivity to the sample size can affect a substantial number of research results.

Table 9

Percentages of results where the conclusion changed as the sample size increased, using Wilcoxon test

4.2.2 Interpreting effect sizes

In this subsection, the measure r for the effect size is added to the study, as defined in Equation (16), in Section 2. The effect size will be calculated for all tests done in order to measure the relevance of the difference between the classifiers’ accuracies, for each dataset. The higher the effect size, the greater the manifestation of the phenomenon (difference) in the population.

Two analyses will be conducted. The first one investigates the behavior of the effect size with an increase in sample size, to compare it with the behavior of the p-value for the same increase. The second one investigates the frequency of the so-called special cases (where there is a disagreement between the p-value and the effect size) in the results for the Wilcoxon test.

The first analysis refers to Table 9, where the first row shows the percentages of results that changed from non-significant to significant as the sample size increased. The second row shows the percentage of results where a small or insignificant effect size (r) changed to a medium or large one as the sample size increased. For example, with the increase of the sample size from 10 to 20, 14.52% of the non-significant cases became significant and 5.03% of the cases with small or insignificant effect size became cases with medium or large effect size.

As shown in Table 9, the percentage of tests where the effect size was upgraded to medium or large was considerably smaller than the percentage of results where the p-value was upgraded to significant, for all three types of sample size increase. Note that, despite not being directly sensitive to the sample size, the effect size has increased or decreased with larger sample sizes, since the estimates of sample mean and variance for larger samples were different from the estimates for smaller samples.

To illustrate this difference between the behaviors of the p-value and the effect size as the sample size increased, we refer again to the comparison between SVM and 1-NN in the ‘Contraceptive Method Choice’ dataset. With sample size 10, the p-value was higher than 0.05, but with sample sizes 20 and 30, it was smaller than 0.05. However, the effect size did not qualitatively change as the sample size increased. The r value was $0.39$ , $0.45$ and $0.3660$ for sample sizes 10, 20 and 30, respectively—all of them a medium effect size.

The second analysis evaluates the frequencies of the special cases, where there is a disagreement between the p-value and the effect size. In the experiments with the Wilcoxon test, similarly to the t-test, cases were found where the difference between the classifiers’ accuracies is statistically significant, although the effect size is small. Cases were also found where the difference is not significant but the effect size is medium. There are also (as expected) cases where these two measures agree: with statistical significance and medium or large effect size, or with no statistical significance and with small or insignificant effect size.

Turning again to our example scenario (comparing SVM and 1-NN in the ‘Contraceptive Method Choice’ dataset), for the sample sizes 20 and 30, the p-value and r measures agree: there is statistical significance and the effect size is medium. For the sample size 10, the difference between the classifiers’ accuracies is not statistically significant, but the effect size is medium, which is considered as a special case.

Figure 4 shows the behavior of the p-value and effect size for each sample size. The shadowed areas represent special cases, and each point represents a test result. Each curve refers to a different sample size. For example, each point in the curve made of circles represents a test conducted with sample size 10 and the points in the shadowed area represent cases with no statistical significance and medium effect size.

Figure 4

Representation of the p-values for Wilcoxon tests applied to samples sized 10, 20 and 30, and the respective values of effect size—the cases in the shadowed areas represent the special cases

Note that, for the three sample sizes, in most cases the p-value and the effect size agree. These are cases where: the p-value is statistically significant (p-value < 0.05) and the r measure indicates a medium or larger effect size ( $r \geq 0.3$ ), or the p-value is not significant (p-value $\geq$ 0.05) and the r measure indicates a small or insignificant effect size ( $r<0.3$ ). However, for the three sample sizes, special cases were also found, where these measures do not agree. These special cases are in the shadowed area of Figure 4. For sample sizes 10 and 20, in some cases the Wilcoxon test’s p-value was not significant, although the r measure for such cases indicated a medium effect size. For sample size 30, some cases were found where the test’s result gave evidence to reject the null hypothesis (p-value < 0.05), although the effect size was small.

Table 10 shows, for each pair of classifiers and each sample size, the percentages of Wilcoxon test results for each of two types of special cases: (a) with no statistical significance and a medium effect size and (b) with statistical significance and a small effect size. Note that cases without statistical significance and medium effect size were found only for sample sizes 10 and 20. On average over the 15 pairs of classifiers, this type of special case was found in $15.72\%$ and $1.08\%$ of the results for sample sizes 10 and 20, respectively. For sample size 10, the percentage was over 20% for 5 pairs of classifiers.

Table 10

Percentage of datasets with no statistical significance in the Wilcoxon test and with a medium effect size, and percentage of datasets with statistical significance and small effect size, for each pair of classifiers, per sample size

Table 10 also shows that cases with a statistically significant p-value and small effect size were found only for samples with size 30. With this sample size, this type of special case was observed for all pairs of classifiers and observed on average for $7.0\%$ of the Wilcoxon test results.

It should be noted that, similarly to the t-test results, the sample size 20 led to the smallest percentage of special cases for the Wilcoxon test results.

4.2.3 Evaluating the power of the test

Now, we evaluate the power of the Wilcoxon test, which is calculated based on sample simulation and considering that the actual value of the difference between the classifiers’ accuracies is equal to the observed difference.

First, we will analyze the power of the test for our example scenario of comparing SVM and 1-NN in the ‘Contraceptive Method Choice’ dataset. As reported in Table 11, the p-value was at $0.08$ , $0.004$ and $0.004$ , and the r effect size was of $0.39$ , $0.45$ and $0.36$ , for sample sizes 10, 20 and 30, respectively. For sample size 10, the difference between the classifiers’ accuracies was not significant, although it had a medium effect size. The power of the test was $0.45$ , that is, a low power (the recommended value should be over $0.8$ ), suggesting that the test was not applied in the ideal scenario. That is, the sample is too small. For sample sizes 20 and 30, the Wilcoxon test’s p-value is statistically significant, whilst the r measure indicated a medium effect size. To reinforce these conclusions, the power of the test was high for both sample size 20 (power $0.84$ ) and size 30 ( $0.88$ ). This indicates that the test has a high probability of rejecting the null hypothesis when it is actually false.

Table 11

Power of the Wilcoxon test, p-value and effect size, when comparing SVM’s and 1-NN’s accuracies in the ‘Contraceptive Method Choice’ dataset

The following analyses use the same four groups of results defined in Subsection 4.1.3. Recall that Groups 1 and 2 have agreeing results between the Wilcoxon’s test p-value and the effect size measure r, whilst Groups 3 and 4 are special cases. Figure 5 shows, for each group, the boxplots for values of the power of the Wilcoxon test, for each sample size (10, 20 and 30).

Figure 5

Boxplots for the power of the Wilcoxon test, per group and sample size

Figure 5 shows that in Group 1, with statistical significance and medium or large effect size, in most cases the test had a high power. In this scenario, the three measures reinforce each other: the high power of the test gives credibility to the test’s result, indicating a statistically significant difference between the classifiers’ accuracies, and the effect size indicates that the magnitude of this difference is relevant. However, for this same group, we observe some cases with a low power, with a higher frequency for sample size 10 than for size 30. This is expected, since the power of the test tends to increase as the sample size increases. In this scenario, although the p-value and the effect size agree, a low power indicates that the Wilcoxon test is not reliable.

Group 2 presents an opposite situation. The Wilcoxon test did not reject the null hypothesis and the effect size is small, agreeing with the test’s result. However, the power of the test is low, indicating a low probability of the test rejecting the null hypothesis when it is false.

For Group 3, where the test did not reveal statistical significance but the effect size was medium or large, the power of the test was low. This suggests that the test is unreliable since it has a low probability of rejecting the null hypothesis when the real difference between the classifiers’ accuracy is equal to the sample difference. In these cases, a large effect size and low power of the test point to the fact that the sample size was too small, failing to provide evidence to reject the null hypothesis.

The power of the test was also low in Group 4, where the Wilcoxon test revealed a statistically significant difference between the classifiers’ accuracies, but the effect size was small or insignificant. In such cases, the result of the test is questionable, since it had a low power and small effect size.

As previously observed in the analysis of the Student t-test, the results for the Wilcoxon test also show that all the cases with a high power ( ${>}0.8$ ) belong to Group 1, as seen in Figure 5. That is, the cases with a high power had statistical significance and a medium or large effect size.

It should be pointed out that the largest differences in the sample accuracies belong to Group 1. As expected, for the cases with large observed differences, the test had statistical significance, the effect size was medium or higher and the power of the test was high. However, it is not only in cases with large sample differences that these results are observed. In Group 3, for example, even with the small sample differences, the effect size was medium or large. As for Group 4, even with small sample differences, the Wilcoxon test revealed statistical significance, indicating a difference between the classifiers’ accuracies.

Group 4 has a sample standard deviation slightly larger than the one for Group 3. Despite this and both groups having too small medians and similar distributions, the Wilcoxon test found evidence to reject the null hypothesis in Group 4, but was not able to reject it in Group 3. The difference in these results may be explained by the sample sizes. Although the standard deviation is slightly higher in Group 4, the sample size is also higher in Group 3, which led to these results.