Recently I was reading a bunch of articles on computational reproducibility in scientific research including one from my fellow blogger Grant Jacobs of Code for life blog. The term reproducibility in computational science is not monolithic - what seems essential for one scientific domain may not matter to another. Finding out what it takes is a difficult, but necessary effort. According to Randy LeVeque, a prominent mathematician
Latest series of debate originated from an article in journal Science written by Jill P. Mesirov on accessible reproducible research in computer science. The center of argument was Jill's own experience with GenePattern-Word RRS system with intention to universalize the described approach. After reading the whole paper I found nothing novel about the kind of reproducibility Jill is proposing in the article, in fact there are plenty of ongoing efforts to support the computational reproducibility by developing niche markup languages and associated tools. Many of these efforts such as SBML and CellML are well in advanced stage, and again most of journals now demand their perspective authors to submit their data and model in domain specific formats. I can't see why these efforts are not a step further towards accessible computational reproducibility. They are totally qualified as mean to achieve the computational reproducibility. Suggestions such as computational research need a paradigm shift or calls out for a new model for the way we publish our results is already in place, may be in primitive form but we are coming there. We talk and talk but never appreciate what communities are doing, this new drive for reproducibility evangelism is faulty and incomplete unless this gives a due consideration for ongoing community wide efforts to tackle this issue.
From this whole debate I was forced to ask few questions,
1. Computational reproducibility is important but to what extent? Reproduction and repetation of scientific protocol is never a straightforward affair. By no means scientific reproducibility should be considered as replacement for scientific integrity and learning. It also reminds me an argument presented by James Bassingthwaighte who thinks that computational reproducibility might be overrated. According to James
Although this is not very universal argument but I am sure readers are well aware about the fact that debugging can be a rewarding experience when you are learning the programing language. Debugging in computational protocols depends on fine tuning of parameters which requires a close understanding of algorithm in use. Same genome analysis pipeline may require different sets of BLAST parameters for the unlike sequence families. This kind of learning is possible only when you challenge the user to come up with a better solution.
(Check out the full presentation below, it is worthwhile)
2. Are development process (for computer softwares, pipeline, models) and end product "reproducible research" are two different things? They look slightly unrelated but they depend on each other. Employing best practice such as use of domain specific standards (languages and guidelines) along with software design principles in the development process results in provenance-aware computer softwares, pipeline and models. Provenance refer to the process of tracing and recording of the origins and evolution of the data. The analysis workflow or pipeline itself can be captured by process definition or workflow description languages while the data remains seated in the wrapper of domain specific markup languages completing the full circle of provenance-aware tools. You want total reproducibility, do this I promise nothing will come in between. Unfortunately its hard to implement most or part of above suggestion, because when it comes to science getting things done (GTD) is major concern than quality of codes we write. Workflow solutions like Taverna, Galaxy, KNIME, Keplar, Pipeline Pilot, Inforsense KDE are designed to track the trace and history of scientific data and analysis workflow in the same way I suggested above. These tools are available for a long time and scientist mostly failed to embrace these ideas and tools. Why? For the same reasons they don't embrace the very idea of web 2.0. Because in science advertisement of scholarship (read it peer reviewed publications) is more important than scholarship itself (I call it the learning experience).
3. What is the role of standards and guidelines? In last few years scientific community has seen a flurry of activity over standards and guidelines development. Are they any good for reproducibility? Yes, indeed. Let me explain this with one example, can you understand the essence of the image published in a peer reviewed scientific article without reading the following text? Well answer is pretty no. Albeit it might be easy to understand the circuit diagram published in a electrical book just because most of notation of circuit schematics are well standardized. The people in scientific community have realized this problem and result is a community driven effort Systems Biology Graphical Notation (SBGN) to standardize the graphical notations in biology. Although SBGN is in its early stage it can play an important role the way we communicate knowledge in biology. Similary in machine learning and data mining community PMML (Predictive Model Markup Language) is doing a fantastic job by enabling users to reuse, post-process, and utilize data mining models irrespective of the tool that generated them. In my humble opinion, notion of putting the reproducibility into computational code is just magical unless we have better underlaying standards and guidelines to support this.
4. What we can learn from Unix and Open source model? Using a "make" inspired computaional reproducibility systems is not new. "make" is a populatr utility program for automatically building executable programs and other non-source files from source code. In past "make" based tools such as SCons has been used as platform for reproducible research in scientific computing. Secondly scientific community should avoid any reliance on proprietary softwares, they make a big gap in reproducibility workflow due to closed source nature.
Closing thoughts
The troubles of computational reproducbility are primarily those of ideology and practice. This may be a slightly controversial assertion, but I would argue that technically reproducibility in computational biology is not a big issue. The adoption of reproducible research practice has been slow. The focuse of reproducbility related discussions in computaional biology should be more on how to inspire the scientific communities to embrace a better reproducible research.
Scientific and mathematical journals are filled with pretty pictures these days of computational experiments that the reader has no hope of repeating. Even brilliant and well intentioned computational scientists often do a poor job of presenting their work in a reproducible manner. The methods are often very vaguely defined, and even if they are carefully defined, they would normally have to be implemented from scratch by the reader in order to test them.
Latest series of debate originated from an article in journal Science written by Jill P. Mesirov on accessible reproducible research in computer science. The center of argument was Jill's own experience with GenePattern-Word RRS system with intention to universalize the described approach. After reading the whole paper I found nothing novel about the kind of reproducibility Jill is proposing in the article, in fact there are plenty of ongoing efforts to support the computational reproducibility by developing niche markup languages and associated tools. Many of these efforts such as SBML and CellML are well in advanced stage, and again most of journals now demand their perspective authors to submit their data and model in domain specific formats. I can't see why these efforts are not a step further towards accessible computational reproducibility. They are totally qualified as mean to achieve the computational reproducibility. Suggestions such as computational research need a paradigm shift or calls out for a new model for the way we publish our results is already in place, may be in primitive form but we are coming there. We talk and talk but never appreciate what communities are doing, this new drive for reproducibility evangelism is faulty and incomplete unless this gives a due consideration for ongoing community wide efforts to tackle this issue.
From this whole debate I was forced to ask few questions,
1. Computational reproducibility is important but to what extent? Reproduction and repetation of scientific protocol is never a straightforward affair. By no means scientific reproducibility should be considered as replacement for scientific integrity and learning. It also reminds me an argument presented by James Bassingthwaighte who thinks that computational reproducibility might be overrated. According to James
•Better ideas will emerge when other investigators attack a model (hypothesis) and attempt to improve on it.
•Redevelopment is the best way to penetrate a model or hypothesis –i.e. no waste of effort!
This-is the real test of reproducibility of the original authors’ views
-really validates it by doing it totally independently
-makes it less likely to incorporate the original’s errors
Copy-cat errors are like mutations in a phylogenetic sequence, so it’s better to start at the beginning each time.
Although this is not very universal argument but I am sure readers are well aware about the fact that debugging can be a rewarding experience when you are learning the programing language. Debugging in computational protocols depends on fine tuning of parameters which requires a close understanding of algorithm in use. Same genome analysis pipeline may require different sets of BLAST parameters for the unlike sequence families. This kind of learning is possible only when you challenge the user to come up with a better solution.
(Check out the full presentation below, it is worthwhile)
2. Are development process (for computer softwares, pipeline, models) and end product "reproducible research" are two different things? They look slightly unrelated but they depend on each other. Employing best practice such as use of domain specific standards (languages and guidelines) along with software design principles in the development process results in provenance-aware computer softwares, pipeline and models. Provenance refer to the process of tracing and recording of the origins and evolution of the data. The analysis workflow or pipeline itself can be captured by process definition or workflow description languages while the data remains seated in the wrapper of domain specific markup languages completing the full circle of provenance-aware tools. You want total reproducibility, do this I promise nothing will come in between. Unfortunately its hard to implement most or part of above suggestion, because when it comes to science getting things done (GTD) is major concern than quality of codes we write. Workflow solutions like Taverna, Galaxy, KNIME, Keplar, Pipeline Pilot, Inforsense KDE are designed to track the trace and history of scientific data and analysis workflow in the same way I suggested above. These tools are available for a long time and scientist mostly failed to embrace these ideas and tools. Why? For the same reasons they don't embrace the very idea of web 2.0. Because in science advertisement of scholarship (read it peer reviewed publications) is more important than scholarship itself (I call it the learning experience).
3. What is the role of standards and guidelines? In last few years scientific community has seen a flurry of activity over standards and guidelines development. Are they any good for reproducibility? Yes, indeed. Let me explain this with one example, can you understand the essence of the image published in a peer reviewed scientific article without reading the following text? Well answer is pretty no. Albeit it might be easy to understand the circuit diagram published in a electrical book just because most of notation of circuit schematics are well standardized. The people in scientific community have realized this problem and result is a community driven effort Systems Biology Graphical Notation (SBGN) to standardize the graphical notations in biology. Although SBGN is in its early stage it can play an important role the way we communicate knowledge in biology. Similary in machine learning and data mining community PMML (Predictive Model Markup Language) is doing a fantastic job by enabling users to reuse, post-process, and utilize data mining models irrespective of the tool that generated them. In my humble opinion, notion of putting the reproducibility into computational code is just magical unless we have better underlaying standards and guidelines to support this.
4. What we can learn from Unix and Open source model? Using a "make" inspired computaional reproducibility systems is not new. "make" is a populatr utility program for automatically building executable programs and other non-source files from source code. In past "make" based tools such as SCons has been used as platform for reproducible research in scientific computing. Secondly scientific community should avoid any reliance on proprietary softwares, they make a big gap in reproducibility workflow due to closed source nature.
Closing thoughts
The troubles of computational reproducbility are primarily those of ideology and practice. This may be a slightly controversial assertion, but I would argue that technically reproducibility in computational biology is not a big issue. The adoption of reproducible research practice has been slow. The focuse of reproducbility related discussions in computaional biology should be more on how to inspire the scientific communities to embrace a better reproducible research.
Original article is
available at Fisheye Perspective blog. Stay
tuned for more posts and subscribe the RSS feed.
