Statistical Version of 100 Year War

After 100+ years of being silent on the inadequacies of the statistic behind many “statistically significant” conclusions, the ASA published a new statement harshly criticizing p-values online last week. Here’s a link for those who are interested, but a short synopsis follows: http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108

The ASA’s actual statement starts on page 8 and includes the following statements:

“Researchers often wish to turn a p-value into a statement about the truth of a null hypothesis, or about the probability that random chance produced the observed data. The p-value is neither.” Ouch.

“Practices that reduce data analysis or scientific inference to mechanical “bright-line” rules (such as “p < 0.05”) for justifying scientific claims or conclusions can lead to erroneous beliefs and poor decision-making.” MORE OUCH!!!

t-test

SOOO what are we to use???  ASA suggests statisticians to “…supplement or even replace p-values with other approaches…” including “…confidence, credibility, or prediction intervals; Bayesian methods; alternative measures of evidence, such as likelihood ratios or Bayes Factors…”

I’ve selectively bolded the ASA’s suggestion to use confidence intervals, which is the first on their list.  Most auditors are more than familiar with confidence intervals due to extensive experience conducting statistical sampling.   But the inclusion of Bayesian models and Bayes Factors stirs up a long debate between Bayesian approaches to probability theory and frequentist frameworks for probability theory.  I equate their inclusion in ASA’s suggestion to be the equivalent of a 9th round uppercut to the chin in the proverbial boxing match between the Reverend Thomas Bayes and Sir Ronald Fisher decades after both their deaths!