{"id":2225,"date":"2016-03-31T12:28:46","date_gmt":"2016-03-31T15:28:46","guid":{"rendered":"http:\/\/blog.scielo.org\/en\/?p=2225"},"modified":"2017-02-09T11:41:52","modified_gmt":"2017-02-09T13:41:52","slug":"reproducibility-in-research-results-the-challenges-of-attributing-reliability","status":"publish","type":"post","link":"https:\/\/blog.scielo.org\/en\/2016\/03\/31\/reproducibility-in-research-results-the-challenges-of-attributing-reliability\/","title":{"rendered":"Reproducibility in research results: the challenges of attributing reliability"},"content":{"rendered":"

By Lilian Nassi-Cal\u00f2<\/strong><\/p>\n

\"Photo:<\/a>

Photo: Andr\u00e9 Occenstein<\/a>.<\/p><\/div>\n

One of the pillars of scientific research combines the trustworthiness of scientists and the reliability of results, which, in turn, support the hypothesis to be proved. Just as the lack of ethics in experimentation and scientific publication, the lack of reproducibility is considered a serious failure that contributes to jeopardize the credibility of science as a whole.<\/p>\n

Studies indicate, however, that more than half of the experiments involving clinical trials of new drugs and treatments are irreproducible. John Ioannidis at Stanford University, US, goes on saying that most of the search results is actually false. Ioannidis is the author of a mathematical model that predicts that the smaller the sample and less stringent are the experimental methodology, definitions, outcomes and statistical analysis, the greater the probability of error. Furthermore, studies that hold financial and other interests or of great impact are also more prone to false results.<\/p>\n

The “science hierarchy” that positions the Exact Sciences at the top, the Humanities at the base and the Biological Sciences between them both has over 200 years, says Daniele Fanelli1<\/sup>, then a researcher at the University of Edinburgh, UK (he is currently senior researcher at Stanford University, US). His study correlates the areas of knowledge, the proportion of positive results and the reliability of results based on the rigor employed to test the author’s hypothesis, held with analysis of over two thousand articles in all disciplines. His results, however, confirm the status of the Social Sciences against numerous arguments that it is a rather subjective field. Fanelli shows that when scientific approach is used, its degree of reliability is located close to that of the Natural Sciences.<\/p>\n

In this controversial scenario comes a study called Reproducibility Project: Psychology2<\/sup> aimed to evaluate the reproducibility of 100 research articles in Psychology, started in 2011 and completed in 2015, and moved by allegations of fraud and failed statistical analysis in classic studies on Psychology. The results, reported in Nature in 20153<\/sup> show that only 39 of them could be reproduced. The results of this study, however, are not absolute, and there are several nuances ranging from “virtually identical” to “somewhat similar” and “not at all similar”. Among the 61 studies that failed to be reproduced, scientists have classified 24 of them as presenting results “moderately similar” to the original experiment, but were disapproved by not achieving statistical significance, a necessary criteria to be considered a successful replication.<\/p>\n

This result would lead to the conclusion that Psychology is not a reproducible science. However, areas such as Cancer Biology, and studies on new drugs have reproducibility rates even lower, according to Fanelli, who considers the result on the Psychology study quite acceptable. The teams that led reproducibility tests were not always available to the same experimental conditions and certainly not the same as the original study patients. This certainly contributes to the low reproducibility of studies.<\/p>\n

Brian Nosek, a social psychologist and head of the Center for Science in the US Open, leader of the Reproducibility Project, worked directly with about 270 employees in the replication of psychology studies. As Fanelli, he reported to Nature4<\/sup> that there is no safe way to state that a certain article is reliable or not from this study. It may be that the original or replication are flawed or that there are substantial differences between them to allow for proper assessment. Nosek stresses that the goal of the Reproducibility Project is simply not vouch for how many articles are reliable, but warn of the publication of results that do not stand up to more detailed scrutiny and quantitatively evaluate the present bias in publications in psychology. He believes that only 3% of the funds for the research were used in evaluation of this nature, would make a huge difference.<\/p>\n

As in other disciplines, it is known that it is not only methodological rigor or statistical significance at stake to approve an article for publication. Periodicals want to attract the attention of your readers to preferentially publish positive or controversial results, sometimes validated by statistical tests handpicked to meet the author’s needs. This is particularly common in biomedicine, and is going on a similar initiative, the Reproducibility Project: Cancer Biology. It is worth noting that its implementation faced in mid-2015 difficulties accessing the original data of the reviewed studies and has not yet been completed.<\/p>\n

However, the area of psychology researchers resumed the study of the Reproducibility Project: Psychology and concluded that there is insufficient evidence to doubt the credibility of the publications, according to Daniel Gilbert, a psychologist at Harvard University, USA, and one of the authors of the review recently published in Science5<\/sup>. A response6<\/sup> published in the same issue of the journal, however, disputes the review claiming that is based on selective assumptions.<\/p>\n

Gilbert defends the reliability of psychology studies and claims that they are as reproducible as any other area. Moreover, in his view, the percentage of results considered confirmed by Reproducibility Project (39%) is of the same order of magnitude than would be expected by chance, even if the original study were true. Analyzing the experimental protocols of the project, it is clear that each study was played only once, showing low statistical significance to confirm or not the original results. In fact, an article published in February this year in PLoS7<\/sup> reassesses statistical tests of the Project, and concludes that about one third of replication is inconclusive.<\/p>\n

The controversy about the attempts to test and certify the reproducibility of scientific studies is viewed with optimism by Nosek and other scientists, it points out that transparency in scientific methodology and statistical verification of the results is crucial in each study. As for contestatory study Gilbert and colleagues, Nosek believes that it cannot be assessed as definitive.<\/p>\n

An article about David Allison authoring Reproducibility studies, Department of Biostatistics, School of Public Health, University of Alabama, USA, and colleagues, was published in Nature in February this year8<\/sup>. In the trial, the authors assess how science is subject to errors and to what extent it autocorrects. Although many fraudulent items or methodology failed to emerge and find themselves portrayed, this, unfortunately, is not the general rule. “Consult a statistician after completing an experiment is how to perform an autopsy. You can perhaps find the cause of death of the experiment, “said the statistician Ronald Fisher, who died in 2015. In his opinion, post publication reviews also are post mortems because attest that the studies were conducted flawed methodology and validated by statistical tests also flawed, but little can be done at this stage.<\/p>\n

In addition to psychology, studies in economics are being evaluated as the reproducibility. An article published in Science9<\/sup> in early March reported a project to replicate 18 studies in economics published in two renowned journals between 2011 and 2014. The researchers found that 11 studies could be reproduced, that number rose to 14 when different criteria were used to evaluate reproducibility.<\/p>\n

According to Nosek, these results do not necessarily indicate that studies in economics are more reproducible than those of psychology, mainly because the number of studies in the first case was smaller and concentrated on studies with simple relationships. In the opinion of John Bohannon, correspondent and collaborator of Science, most studies could not be reproduced employed as statistical test a p-value below 5% as significant. According to the author, although many are in accordance with the test fragility, few are willing to discuss it. Some authors who have had their unplayed results stated that the study methodology was careful, correct and transparent, but do not agree that this means that the original test was a false positive. “We believe it is more accurate to interpret the failure to replicate our study as a treatment failure”10<\/sup>.<\/p>\n

Researchers who were not involved with any of reproducibility projects believe that these different results are inherent to the social sciences, because the population is very heterogeneous. The proposed solution, which serves to all areas of knowledge, would be based on findings in multiple studies on the same subject, to enhance its credibility.<\/p>\n

Notes<\/h3>\n

1. FANELLI, D. \u201cPositive\u201d Results Increase Down the Hierarchy of the Sciences. PLoS ONE<\/em>. 2010, vol. 5, n\u00ba 4, e10068. DOI: 10.1371\/journal.pone.0010068<\/a><\/span>. Available from: http:\/\/www.plosone.org\/article\/info%3Adoi%2F10.1371%2Fjournal.pone.0010068<\/a><\/p>\n

2. Open Science Collaboration. Estimating the reproducibility of psychological science.Science<\/em>. 2015, vol. 349, n\u00ba 6251, aac4716. DOI: 10.1126\/science.aac4716<\/a><\/span>. Available from http:\/\/osf.io\/ezcuj\/wiki\/home\/<\/a><\/p>\n

3. BAKER, M. First results from psychology\u2019s largest reproducibility test. Nature<\/em>. 2015. DOI: 10.1038\/nature.2015.17433<\/a><\/span>. Available from http:\/\/www.nature.com\/doifinder\/10.1038\/nature.2015.17433<\/a><\/p>\n

4. BAKER, M. Over half of psychology studies fail reproducibility test. Nature<\/em>. 2015. DOI: 10.1038\/nature.2015.18248<\/a><\/span>. Available from: http:\/\/www.nature.com\/doifinder\/10.1038\/nature.2015.18248<\/a><\/p>\n

5. GILBERT, D.T., et al<\/em>. Comment on \u201cEstimating the reproducibility of psychological science\u201d Science<\/em>. 2016. vol. 351, n\u00ba 6277, pp. 1037. DOI: 10.1126\/science.aad7243<\/a><\/span>. Available from: http:\/\/science.sciencemag.org\/content\/351\/6277\/1037.2<\/a><\/p>\n

6. ANDERSON, C.J., et al<\/em>. Response to Comment on \u201cEstimating the reproducibility of psychological science\u201d. Science<\/em>. 2016, vol. 351, n\u00ba 6277, pp. 1037. DOI: 10.1126\/science.aad9163<\/a><\/span>. Available from: http:\/\/dx.doi.org\/10.1126\/science.aad9163<\/a><\/p>\n

7. ETZ, A. and VANDEKERCKHOVE, J. A Bayesian Perspective on the Reproducibility Project: Psychology. PLoS ONE<\/em> 2016, vol. 11, n\u00ba 2, e0149794. DOI: 10.1371\/journal.pone.0149794<\/a><\/span>.<\/p>\n

8. ALLISON, D.B., et al<\/em>. Reproducibility: A tragedy of errors. Nature.<\/em> 2016, vol. 530, n\u00ba 7588, pp. 27-29. DOI: 10.1038\/530027a<\/a><\/span>. Available from: http:\/\/www.nature.com\/news\/reproducibility-a-tragedy-of-errors-1.19264<\/a><\/p>\n

9. CAMERER, C.F. et al<\/em>. Evaluating replicability of laboratory experiments in economics. Science<\/em>. 2016, vol. 351, n\u00ba 6280, pp. 1433-1436. DOI: 10.1126\/science.aaf0918<\/a><\/span>. Available from: http:\/\/science.sciencemag.org\/content\/351\/6280\/1433<\/a><\/p>\n

10. BOHANNON, J. About 40% of economics experiments fail replication survey. Science<\/em>. 2016. DOI: 10.1126\/science.aaf4141<\/a><\/span>. Available from: http:\/\/www.sciencemag.org\/news\/2016\/03\/about-40-economics-experiments-fail-replication-survey<\/a><\/p>\n

References<\/h3>\n

ALLISON, D.B., et al<\/em>. Reproducibility: A tragedy of errors. Nature.<\/em> 2016, vol. 530, n\u00ba 7588, pp. 27-29. DOI: 10.1038\/530027a<\/a><\/span>. Available from: http:\/\/www.nature.com\/news\/reproducibility-a-tragedy-of-errors-1.19264<\/a><\/p>\n

ANDERSON, C.J., et al<\/em>. Response to Comment on \u201cEstimating the reproducibility of psychological science\u201d. Science<\/em>. 2016, vol. 351, n\u00ba 6277, pp. 1037. DOI: 10.1126\/science.aad9163<\/a><\/span>. Available from: http:\/\/dx.doi.org\/10.1126\/science.aad9163<\/a><\/p>\n

BAKER, M. First results from psychology\u2019s largest reproducibility test. Nature<\/em>. 2015. DOI: 10.1038\/nature.2015.17433<\/a><\/span>. Available from http:\/\/www.nature.com\/doifinder\/10.1038\/nature.2015.17433<\/a><\/p>\n

BAKER, M. Over half of psychology studies fail reproducibility test. Nature<\/em>. 2015. DOI: 10.1038\/nature.2015.18248<\/a><\/span>. Available from: http:\/\/www.nature.com\/doifinder\/10.1038\/nature.2015.18248<\/a><\/p>\n

BEGLEY, C.G.\u00a0and\u00a0ELLIS, L.M.\u00a0Drug development: Raise standards for preclinical cancer research. Nature<\/em>.\u00a02012, vol. 483, 7391, pp. 531-533. DOI: 10.1038\/483531a<\/a><\/span><\/p>\n

BOHANNON, J. About 40% of economics experiments fail replication survey. Science<\/em>. 2016. DOI: 10.1126\/science.aaf4141<\/a><\/span>. Available from: http:\/\/www.sciencemag.org\/news\/2016\/03\/about-40-economics-experiments-fail-replication-survey<\/a><\/p>\n

CAMERER, C.F. et al<\/em>. Evaluating replicability of laboratory experiments in economics. Science<\/em>. 2016, vol. 351, n\u00ba 6280, pp. 1433-1436. DOI: 10.1126\/science.aaf0918<\/a><\/span>. Available from: http:\/\/science.sciencemag.org\/content\/351\/6280\/1433<\/a><\/p>\n

ETZ, A. and VANDEKERCKHOVE, J. A Bayesian Perspective on the Reproducibility Project: Psychology. PLoS ONE<\/em> 2016, vol. 11, n\u00ba 2, e0149794. DOI: 10.1371\/journal.pone.0149794<\/a><\/span>.<\/p>\n

FANELLI, D. \u201cPositive\u201d Results Increase Down the Hierarchy of the Sciences. PLoS ONE<\/em>. 2010, vol. 5, n\u00ba 4, e10068. DOI: 10.1371\/journal.pone.0010068<\/a><\/span>. Available from: http:\/\/www.plosone.org\/article\/info%3Adoi%2F10.1371%2Fjournal.pone.0010068<\/a><\/p>\n

GILBERT, D.T., et al<\/em>. Comment on \u201cEstimating the reproducibility of psychological science\u201d Science<\/em>. 2016. vol. 351, n\u00ba 6277, pp. 1037. DOI: 10.1126\/science.aad7243<\/a><\/span>. Available from: http:\/\/science.sciencemag.org\/content\/351\/6277\/1037.2<\/a><\/p>\n

IOANNIDIS, J. P. Why most published research findings are false.\u00a0PLoS Med<\/em>. 2005. DOI: 10.1371\/journal.pmed.0020124<\/a><\/span>. Available from: http:\/\/www.plosmedicine.org\/article\/info:doi\/10.1371\/journal.pmed.0020124<\/a>.<\/p>\n

NASSI-CAL\u00d2, L. Reproducibility of research results: a subjective view<\/em>. SciELO in Perspective. [viewed 06 March 2016]. Available from: http:\/\/blog.scielo.org\/en\/2014\/02\/19\/reproducibility-of-research-results-a-subjective-view\/<\/a><\/p>\n

NASSI-CAL\u00d2, L. Reproducibility of research results: the tip of the iceberg<\/em>. SciELO in Perspective. [viewed\u00a006 March 2016]. Available from: http:\/\/blog.scielo.org\/en\/2014\/02\/27\/reproducibility-of-research-results-the-tip-of-the-iceberg\/<\/a><\/p>\n

Open Science Collaboration. Estimating the reproducibility of psychological science.Science<\/em>. 2015, vol. 349, n\u00ba 6251, aac4716. DOI: 10.1126\/science.aac4716<\/a><\/span>. Available from http:\/\/osf.io\/ezcuj\/wiki\/home\/<\/a><\/p>\n

PRINZ, F., SCHLANGE, T., and ASADULLAH, K. Believe it or not: how much can we rely on published data on potential drug targets?\u00a0Nature Reviews Drug Discovery<\/em>. 2011, vol. 10, n\u00ba 712. DOI: 10.1038\/nrd3439-c1<\/a><\/span>. Available from: http:\/\/www.nature.com\/nrd\/journal\/v10\/n9\/full\/nrd3439-c1.html<\/a><\/p>\n

VAN NOORDEN, R. Sluggish data sharing hampers reproducibility effort. Nature<\/em>. 2015. DOI: 10.1038\/nature.2015.17694<\/a><\/span>. Available from: http:\/\/www.nature.com\/news\/sluggish-data-sharing-hampers-reproducibility-effort-1.17694<\/a><\/p>\n

External link<\/h3>\n

Reproducibility Project: Cancer Biology – <http:\/\/validation.scienceexchange.com\/#\/cancer-biology<\/a>><\/p>\n

 <\/p>\n

\"lilian\"<\/a>About\u00a0Lilian Nassi-Cal\u00f2<\/h3>\n

Lilian Nassi-Cal\u00f2 studied chemistry at Instituto de Qu\u00edmica <\/em>\u2013 USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME\/PAHO\/WHO and a collaborator of SciELO.<\/p>\n

 <\/p>\n

Translated from the\u00a0original in portuguese<\/a>\u00a0by Lilian Nassi-Cal\u00f2.<\/p>\n","protected":false},"excerpt":{"rendered":"

Recently projects have been developed with the aim to reproduce published research results in psychology, biology and economics to verify their reliability. The results indicate different degrees of reproducibility in each area, however, they served to alert the scientific community about how fragile results considered irrefutable can be and reflect on the role of science in self-correcting. …<\/span> Read More →<\/span><\/a><\/span><\/p>\n","protected":false},"author":22,"featured_media":2228,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[3],"tags":[37,64,31,7],"class_list":["post-2225","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analysis","tag-ethics-in-scholarly-communication","tag-reproducibility","tag-research-evaluation","tag-scholarly-communication"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts\/2225","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/comments?post=2225"}],"version-history":[{"count":4,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts\/2225\/revisions"}],"predecessor-version":[{"id":2486,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts\/2225\/revisions\/2486"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/media\/2228"}],"wp:attachment":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/media?parent=2225"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/categories?post=2225"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/tags?post=2225"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}