Is P-value<0.05 Enough? Two Case Studies in Classifiers Evaluation

A common tool used in the process of comparing classifiers is

the statistical significance analysis, performed through the hypothesis test.

However, there are many researchers attempting to obtain statistical significance through a blinding evaluating of the p-value<0.05 condition, ignoring

important concepts such as the effect size and statistical power. This work

highlights possible problems caused by the misuse of the hypothesis test and

how the effect size and the statistical power can provide information for

a better decision making. Therefore, two case studies applying Student’s

t-test and Wilcoxon signed-rank test for the comparison of two classifiers

are presented.