By Ken Perrott 18/09/2017


We often tout peer review as the reason for accepting the veracity of published scientific studies? But how good is it really? Does it ever match the ideal picture people have of it? And what about peer review before and after publication – are we neglecting these important stages?

Pre-publication peer review

Here I mean the collective process of evaluating ideas and presentations together with scientific colleagues. It’s great when it happens. Ideas flow and the critiques help prevent mistakes from persisting

This happens during discussion of research proposals and of research results. It happens during preparation of presentations.

But, unfortunately, it does not always happen – in fact, I suspect it may be relatively rare. When scientific reforms were introduced into New Zealand almost 30 years ago I noticed some scientific colleagues became less forthcoming about their ideas and research proposals. An air of competition seemed to destroy the previous cooperation.

Maybe things are better now. Hopefully, there is less completion between individuals and within groups and institutions – although I imagine the competition between institutions will always be a problem. Quite apart from competing for grants humans simply identify with their own groups and fall victim to the “them vs us” problem.

Publication peer review

peer reviewThere is an impression that publication peer review happens only when the paper is submitted to a journal. But I think some of the best reviewing of a draft paper actually comes from colleagues before submission. That is why I strongly appreciated the institutional requirements I experienced that a draft paper be peer-reviewed within the institution before submission.

Unfortunately, not all institutions require this. I sometimes think many universities which don’t require this are taking “academic freedom” too far.

Perhaps some scientists see this as only landing extra work on them – but surely knocking a paper into better shape before submission is beneficial to both authors (getting a better draft)  and institutes (maintaining a reputation with journals).

Then there is the peer review organised by the journal. Many people think that is the only peer review. Just as well it isn’t because it can be very bad.

I am sure many poor quality papers slip through to be published simply because reviewers do not do a good job or spend insufficient time on that job. Personally, my impression of reviewers and journals drop when I see reviewers comments indicating a lack of attention or responsibility. Even worse, when I have had a paper accepted by an editor saying the reviewers had no comments I seriously questioned the quality of the journal and the advisability of submitting to it in future.

Still, when an author gets conscientious reviewers and comments indicating the paper has been read carefully an author can’t help but be appreciative – even if it means more work knocking the paper into shape.

As a reviewer, I always attempted to do a thorough job – even if it meant producing an over-long and detailed report. I once a received, via an editor, a response from an author I had reviewed expressing appreciation of the detail so I know such attention to detail is worthwhile.

I think most scientific authors will have occasionally faced the problem of brief or perfunctory reviewing of their submitted papers and can, therefore, understand the feelings behind that note.

Post-publication peer review

This is hardly ever considered. Once published the authors move on – their job is done. Readers also tend to be very accepting of published papers – after all peer review means that the paper’s findings must be trustworthy.

But this is obviously not the case. I think the slogan “reader beware” applies just as much to the scientific literature as it does to the news media. The reader should not automatically accept reported findings or conclusions as correct – just because the paper was peer-reviewed. They should do their own due diligence, consider all papers critically and avoid automatic acceptance.

Formal post-publication peer review can occur – but it is not as common as it should be. Some online journals provide space for readers comments. Helpful to the author but not adequate for proper evaluation.

The best post-publication peer review comes from published critiques because they become part of the established literature and available to anyone following up a subject or reviewing a field. Some journals provide space for shorter critiques of this sort – not requiring these authors to present new and original data but simply critique what has been published. Of course, despite the lower requirements, such critiques should undergo their own peer review consistent with the policies of the journal.

The ethics of post-publication review

This is sore point for me – having had an editor recently refuse to consider a critique of mine (see Fluoridation not associated with ADHD – a myth put to rest).

Surely there is a moral obligation for a journal, and its editor, to consider submissions of critiques of paper they have published? This is the obvious place for a critique – and the journal can normally then offer the right of reply to the original authors.  The writer of a critique should not have to search out an alternative journal – especially as the lack of new data or new research in a critique makes its acceptance by an alternative journal problematic. Nor should the original authors be denied an automatic right of reply which can be provided by the original journal.

Authors of a critique can face obstacles like the cost of publication. An original paper may be published in a journal which extracts publication fees from the author. It is the original authors decision whether or not to publish in such journals. But it seems unethical to expect the submitter of a critique to pay such fees. That puts a financial hurdle in the way of proper scientific peer-review. The original authors’ institution may be prepared to cover the cost of publication but institutions are unlikely to financially cover critiques in the same way.

The other obstacle is, of course, the attitude of editors. It is surely just common sense that critiques should undergo the normal peer review but when journals or editors refuse outright to even consider a critique, to not even enable it to undergo peer review, then that is ethically wrong.