For the second time in a week I have removed the word “significant” from a draft manuscript written by a colleague of mine in clinical medicine. In a significantly p’d I wrote about the myth of significance – that is about the ubiquitous use of the term “significant” in the medical literature to mean a specific probability incorrectly rejecting the hypothesis that two things (eg two treatments) are the same (you may need to read that twice). What I pointed out was the “significant” does not mean “meaningful.” Here I want to propose an alternative. But first, I need to discuss two major problems with the term.
Where common is not specific
In my experience the common usage of “significant” to mean important is the normal interpretation of the word in the science literature even by many medically trained people and sometimes the authors of articles themselves.
The tyranny of p<0.05
When the maths wiz Ronald Fisher talked about significance (in an agricultural journal not a medical one!) he used 0ne in 20 (p<0.05) as an acceptable error rate in agricultural field trials so that trials did not have to be repeated many times. That p<0.05 has taken on almost magical proportions (‘scuse the pun) in the medical literature is scary and shameful. I don’t want to delve into all that now. If you want to, a starting point maybe the Nature article here.
I propose that in all scientific literature that authors replace the term “significant” with the phrase “beyond reasonable doubt” and that they only be allowed to publish the article if in the methods section they define what p value they choose to represent “beyond reasonable doubt” and they defend why they have chosen this value and not another. “Beyond reasonable doubt” is a term used in the New Zealand judicial system where those charged with a crime are presumed innocent (Null hypothesis) until proven otherwise. Perhaps those of us in science could learn something from our lawyer friends.