Post publication peer review – a new way of doing science?

By Grant Jacobs 30/05/2011

I lead off from Carl Zimmer relating how on-line conversation played a role in ’the arsenic bacteria’ publication as an excuse to muse about post-publication peer review.

Reading Carl Zimmer’s update on the ‘arsenic bacteria’ story in Slate last night, while writing my own update, I encountered again the phrase ‘post publication peer review’:

’Redfield and her colleagues are starting to carry out a new way of doing science, known as post-publication peer review. Rather than leaving the evaluation of new studies to a few anonymous scientists, researchers now debate the merit of papers after they have been published.’

On his blog, Carl introduces his Slate article:

Over at Slate, I look at how the online conversation has changed the way scientists do their work.

Post-publication criticism, in all it’s forms,[1] follows any research publication of note or notoriety. Has done for years, decades. I don’t think there is a lot new there.

I personally suspect ‘the arsenic bacteria’ media story may be better viewed as an example of the media making more (read: better!) use of on-line criticism–blogs and all that, too–than it typically does, rather than a change in how science is done. As I wrote last time on this:

There is nothing new going on in scientists writing on blogs, compared to writing on bionet (or its ilk). There might be something new in journalists tapping into this writing, perhaps-?

More to the point, my reading is that those promoting post-publication peer review mean something slightly different by post-publication peer review.

In a nutshell, some people (e.g. at The Future of Scientific Publishing) are suggesting that peer review of research papers be open. In addition to a few selected reviews invited by the journal in question, allow any researcher willing to comment to file a response to the journal that is retained for the record and open to be read by all.

Post-publication peer review, then, is not proposed as ‘a new way of doing science science itself isn’t changing. It’s proposed as a new, additional, element in ‘formal’ scientific publishing.

Informal commentary isn’t new. Papers–good or bad–that attract attention are always debated after they are published.

More formalised commentary directly associated with on-line publication isn’t new either. Technical Comments, Letters to the Editor, Editorials, etc., offered in response to the on-line in-advance publication are the usual business that follows controversial publications.

As far as I am able to make out on short notice, all post-publication peer review seems to seek is to encourage and formalise this general style of review, tying it directly to the research publication. To me personally, I don’t find it ’that’ new: it almost feels a use of internet-based technology rather than something fundamentally new.

(I am in some ways reminded of the correspondence found in some journals, particularly the early scientific publications. I’d be curious to see a science historians comparison of Victorian and present-day science here.)

Putting aside whatever parallels there might, or might not, between how Wolfe-Simon et al’s paper is being published and post-publication peer review, two obvious pragmatic problems for post-publication peer review come to mind:

  1. It’s already hard for journals to find (qualified and experienced) reviewers.
  2. Papers that either are ’obviously sound’ or ’too inconsequential’ aren’t easily going to attract volunteer reviewers.

Researchers actively working in the field will generally make the best informed reviewers, but these people already have strong demands on their time. What incentive is there for them to volunteer peer-review letters?

There may be a reluctance to openly offer criticism for fear of jeopardising careers. (Some suggest that the reviewer can choose to be anonymous, but this also has potential problems.) Other criticisms have been noted, for example that a survey of post-publication peer review revealed that it was a poor predictor of the number of citations the paper attracted (one measure of success of a paper is how often later paper cite it). Another was the potential for political, geographical and other biases in the reviews.[2] Pressure groups can also act to sway views on some research.

Personally, I see room for a mix of approaches. I don’t see post-publication peer review as some ideal solution that will fix all of peer-review’s ills, but that its possibilities are at least worth exploring.

Finally, I think that this perhaps emphases the need for good editors regardless of how the journal is run. Editors, to me, are an issue for science writing, too. But both feel like another topic and another post at this late hour…


(This is what I get for writing in the wee hours…)

I hope it’s clear I’m not trying to ’bash’ Carl! I love his writing. It’s just I kept bumping into this phrase ‘post-publication peer review’ and was finally prompted into mulling about it. After all, that’s what good writes does, doesn’t it? – get the reader thinking.

I should add that my field may make my experience the internet aspect of post-publication criticism atypical of science in general. I’m a computational biologist. Being a computational field, you’d hope it takes on internet-based technology readily, perhaps than science as a whole. Thus, my personal experience might not represent that of science as a whole very well. This only affects the internet-based aspect, of course.


Has a lot been written about ‘the arsenic life’ story? Judge for yourself.

[1] Letters to editors, editorials, ‘news and views’ articles, journal clubs (virtual or physical), discussion on-line, conversations at coffee or conferences, to name a few. Review papers, or chapters (like the one I am currently writing, for that matter), are forms of post-publication debate, too.

[2] You’d think conventional peer-review would also suffer from these, with perhaps the saving grace being that a decent editor would perhaps pick up on problems and bear them in mind.

Other articles on Code for Life:

NASA: science shouldn’t be debated in media and blogs?!

Loops to tie a knot in proteins?

Temperature-induced hearing loss

Honey’s anti-bacterial properties found?

Describe your fantasy institute

0 Responses to “Post publication peer review – a new way of doing science?”

  • My impression of the situation is that the “new” aspect of the process is the speed at which papers are dissected and discussed by interested parties. Where previously feedback on a paper might trickle in over weeks or months it is now received within days, sometimes hours.

  • it is now received within days, sometimes hours

    Assuming it’s a happening thing for a moment, with commenting facilities “live”: by the journals. I don’t see that is a “new way of doing science”, however. Science publication, perhaps. Science communication, yes – provided the outside parties access it and make (proper) use of it.

    (I’m distinguishing the process of generating science data—“doing science”—from reviewing the output.)

    The speed of response isn’t the main aim of PPPR, to my reading. (Anyone is welcome to correct me.) It seems to be aiming to address what proponents perceive as inadequacies in “conventional” peer review.*

    Provided the science is treated properly, the speed of post-publication responses might help aid correct things in media more quickly,** but that’s about science communication and depends on the media using it.

    If nothing else science itself will always take longer as researchers need time to think and discuss, particularly if the topic needs information from more than one area, unless the faults are trivial and striking. (That last has been a factor in controversial papers “since forever”. It’s worth factoring into thinking about this.)

    (FWIW, an example I’m thinking of on the bionet usenet groups from my Ph.D. days occurred within days, being the speed of the internet at that time. The formal publication of it took weeks.)

    * I’ve no doubt there are issues, but you can also argue it’s the better than even more flawed alternatives.
    ** I have ideas on this, something I’ve been meaning to write about. But that’d be another post! 😉