Research Reproducibility

Reproducibility: “the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results…. Reproducibility is a minimum necessary condition for a finding to be believable and informative.”

Replicability: "The ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data are collected."

Report of the Subcommittee on Replicability in Science Advisory Committee, - U.S. National Science Foundation, 2015

Reproducible research (computational reproducibility): Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

Replication: A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods), and completing new analyses.

Terminologies for reproducible research. 2018

• "Reproducible Research in Computational Science" 

  Science 334.6060 (2011): 1226-1227.

• Best practices for computational science: Software infrastructure and environments for reproducible and extensible research

  Computing in Science & Engineering Volume: 14, Issue: 4, July-Aug. 2012

• Computing Workflows for Biologists: A Roadmap

  A guide to computing workflows with the biologist end-user in mind. PLoS Biology. November 2015 https://doi.org/10.1371/journal.pbio.1002303

• Congratulations. Your Study Went Nowhere.

  The New York Times. September 24, 2018. Covers research and publication bias. Links to several studies.

• The economics of reproducibility in preclinical research

  PLoS Biol 13(6): e1002165. doi:10.1371/journal.pbio.1002165

• 1,500 scientists lift the lid on reproducibility

  Nature News, 533(7604), 452.

• A long journey to reproducible results 

  Nature News, 548(7668), 387.

• Nonreplicable publications are cited more than replicable ones

  Science advances 7.21 (2021): eabd1705.

• A primer on the ‘Reproducibility Crisis’ and ways to fix it

  Australian Economic Review 51.2 (2018): 286-300.

• The Irreproducibility Crisis of Modern Science

  National Association of Scholars. 221 Witherspoon Street 2nd Floor, Princeton, NJ 08542-3215, 2018.

• Why Most Published Research Findings Are False

  PLoS medicine 2.8 (2005): e124.

• Estimating the reproducilbility of psychological science

  Science. August 2015 Volume 349 Issue 6251

• Evaluation of excess significance bias in animal studies of neurological diseases

  PLoS Biol 11(7): e1001609. doi:10.1371/journal.pbio.1001609

• No Raw Data, No Science: Another Possible Source of the Reproduciblity Crisis

  Editorial in Molecular Brain, February 2020

• Reproduciblity and reliability of biomedical research: Improving research practice

  UK Academy of Medical Sciences symposium report. October 2015.

• Ten Simple Rules for Reproducible Computational Research

  PLoS computational biology 9.10 (2013): e1003285.

Bishop's “Four Horsemen of Irreproducibility Apocalypse” - Main factors that may lead to irreproducible or false positive research: 

1. Publication bias: Researchers are more likely to write about significant results and journal editors are more likely to accept manuscripts showing statistics significance or an expected result
2. Low statistical power: Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result.
3. P-hacking: Running multiple studies but only reporting those that returned significant results.  Also known as data dredging or fishing expeditions.  
4. HARKing (hypothesizing after results are known): When researchers state or change their hypothesis after results are analyzed. Also known as a post hoc hypothesis

(2018). Checklists work to improve science. Nature, doi: https://doi.org/10.1038/d41586-018-04590-7 ; Bishop, D. (2019). Rein in the four horsemen of irreproducibility. Nature, 568(7753), 435-436.; Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., & Munafò, M. R. (2017). Low statistical power in biomedical science: a review of three human research domains. Royal Society Open Science, 4(2), 160254. ; Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217.

Reproduced from: https://libguides.tulane.edu/reproducibility under a Creative Commons Attribution-NonCommercial 4.0 International License.