Nature’s reproducibility effort: when to get data specialists on board

By Grant Jacobs 27/04/2013 2


Scientific magazine Nature has announced an initiative to aimed at aiding the reproducibility of biological research papers. They’re planning to expand the methods section to cater for this, introducing a checklist and offering a Protocol Exchange site.

I’d like to draw attention to one point ArsTechnica’s coverage of this development noted that has wider impact, outside of this Nature initiative:*

To let certain readers know exactly how likely a given result is, Nature will now provide the authors with a statistician to consult (which, really, they should have arranged for themselves before even writing the paper). Authors will be encouraged to provide the underlying data for any charts or graphs in the paper.

This is too late in the act for the research, although understandable from what Nature is trying to achieve for itself and the relationship with the wider public.

The time to start getting the statistician, the computational biologist or other ‘data’ specialist involved isn’t before writing the paper or even when you start the work, as one commenter there wrote, but when you write the grant application.

You want them involved from the onset so that they might help design a sound project. Hopefully the grant application will fare better too. You want to them to head off problems in the research before they turn into headaches, not to have your data team ‘make do’ with whatever data was collected.

I’d like to think larger projects in, say, genomics have long got their head around this one (we hope), but equally I suspect a lot of smaller projects still try tackle the data as it occurs, which is essentially after-the-fact.

Returning to the Nature initiative, some are questioning the effort it might involve both scientists and reviewers. Personally, I’d rather have more complete methods. It’s very frustrating when an aspect isn’t made clear. It reminds me of a loose saying that a project (or computer program) isn’t complete until it’s properly documented. It also reminds that generally it’s easier—and better—to document as you go along. Lab notebooks** are the start everyone uses but what’s needed is something that communicates to someone not familiar with the project.

On alternative that one commenter mentioned is,

That said, Nature tends to be almost more in the style of magazine articles these days. Its more about the audience than the content. Ideally, you would submit an overview paper to Nature, and refer to a more detailed paper for people who want to dig deeper into it. That way, you (a) get two papers out of it, and (b) you don’t turn off all the people who read Nature that aren’t even remotely in your subfield.

It’s an interesting point but personally I wonder if it’s better done in a more science + science-communication fashion with paper kept as papers, with all the detail they need, with lighter ‘reader’ pieces for those with a less focused interest in, say, an on-line magazine format (or science blogs).

There’s also Nature encouraging researchers to provide the data behind graphs, etc. In the interests of time, I’ll not explore this here (but feel free to share their thoughts on this below if you wish to).

The discussion at ArsTechnica is worth reading if this topic interests you – it’s got a pretty good signal to noise ratio.

Footnotes

It’s worth remembering Nature has an extensive supplements section, should it be needed. The alterations are to the Methods section.

* I have to admit I wonder what they mean by ‘certain readers’.

** Someone is bound to mention open notebooks, so I will here! I’m also reminded of literate programming.


Other articles on Code for life:

Reproducible research and computational biology

External (bioinformatics) specialists: best on the grant from the onset

Developing bioinformatics methods: by who and how

Retrospective–The mythology of bioinformatics

You still have to know how the tools work

Bioinformatics – QC, reproducible, statistical and sequence-oriented


2 Responses to “Nature’s reproducibility effort: when to get data specialists on board”

    • Oh, I think peer review is under constant open peer review 😉

      Joking aside, I think there’s constant comments about peer review – it’s one of those perennial topics. There’s a recent summary of the new and/or upper-end open-access journals at ArsTechnica. It was interesting to see eLife’s idea of presenting one review after the reviewing interacting together on-line. (I’m biased perhaps, because like many of the ideas eLife is using, some of which are similar to or the same as suggestions I’ve made in the past.)

      FWIW, I don’t think expanding the Method section will make life harder for either the writer or the reviewer. In my experience it’s harder to cram stuff in than have space to present it and it’s harder to try figure things out than have them laid out. It’s one reason why I’ve liked journals that give more space in the past, e.g. J Mol Biol.