Motivated by a personal experience of the unorthodox behavior of the publisher Scientific & Academic Publishing Co. (SAP) in 2012 surrounding the levying of article processing charges for a paper written by a Nigerian researcher, the biologist and scientific journalist John Bohannon submitted 304 slightly differing versions of a fictional scientific paper, penned by a non-existent author working in an equally non-existent institution to a selected group of journals which publish articles under golden road Open Access, some of which levy an article processing charge. In 157 cases, the paper which contained fundamental errors in both conception and interpretation, was readily accepted for publication, revealing financial interest as the principle motive behind this decision. These journals have publication addresses in various countries, but addresses in India made up one third of these. In Brazil, the journal Genetics and Molecular Research published by the Foundation for Scientific Research of Ribeirão Preto, and indexed in The Web of Science, JCR and Scopus, fell into this trap. Some of the large commercial publishers, such as Elsevier, also had journals which accepted this fraudulent paper for publication. However, it must be said that ninety eight journals promptly rejected the paper for publication.
The article which tells how this idea came about, the way in which the different papers were created and the choice of journals to which the papers were submitted, as well as the results of the “experiment”, was published in the prestigious journal Science¹ on 4th October, 2013.
Bohannon refers to his article in Science as a “sting in the tail” for Open Access. Nevertheless, just as his fraudulent paper contains methodological flaws and an unbalanced approach under which the experiments were carried out with the substances that were tested and the corresponding controls, his article in Science also contains them. The author does not apply the usual practice of using a control group, which in this case would have been formed by non-Open Access journals. Therefore, the question which remains and which is still unanswered is: How would non-Open Access journals behave when faced with a fraudulent paper of this type ? Another question is that Bohannan only selected Open Access journals which charge article processing fees and many of them already have a reputation as “predators”. Nevertheless, the number of Open Access journals which do not charge fees is very large.
Michael Eisen, a biologist from the University of Berkeley and a researcher at The Howard Hughes Medical Institute nevertheless draws attention, on his own blog, to an article authored by Wolfe-Simon et. al. published in Science in 2011 in which he describes the existence of bacteria that use arsenic instead of phosphorous in the make-up of its DNA (one of the five elements which make up the macromolecule). The article, which describes experiments which are at the very least questionable, in order to put forward a theory that would impact on our “understanding of the science behind the origin of life on Earth” and on matters connected with evolution, was accepted for publication after passing through the journal’s allegedly critical peer-review process. Eisen, therefore, highlights that the precarious nature of the peer-review process is not limited to the Open Access journal universe but also touches on celebrated high impact journals. His explanation for the fact that this paper was accepted for publication lies in the fact that it puts forward a theory that is fairly innovative and attractive.
Immediately after the date of publication of the article in Science, dozens of scientific and amateur blogs, journals and magazines, and social networks around the world reported and / or commented on the results of this study. Some agree that there is an unambiguous relationship between Open Access and the absence of peer review, or a low quality of publications. There are, nevertheless, various aspects which challenge the generalized nature of the conclusions drawn by the author, which were pointed out in the large number of comments which the article received – and has continued to receive since its publication. In this post, we will go into some of these in greater detail.
The Idea
The author describes how the idea emerged to submit a fraudulent paper with innumerable and very obvious flaws to a large number of Open Access journals. A researcher in Nigeria tried to publish an article on culex quinquefasciatus, a mosquito that carries West Nile virus and other pathogens in the Open Access journal Public Health Research, believing that it was a journal which did not charge author fees since there was no mention of this on the journal site. However, after her paper was accepted for publication, she was asked to pay a $150 publication fee which was already discounted by 50% because she was an author based in Nigeria. Given the inherently twisted difficulties in an African country in making a payment in foreign currency, and thanks to the intervention of a colleague in the USA who made the payment (which had been further reduced to $90), her paper was published.
It was this fact that made Bohannon look into SAP, the publisher of Public Health Research, where he discovered a whole series of irregularities such as plagiarism, and a fictitious address and editorial body which led him to decide to submit a paper of his own to a SAP journal, and in order to get the lay of this publishing landscape, he took the decision to “replicate the experiment across the entire Open Access world”.
Methodology
The Who’s Who of Open Access journals used by Bohannon was the Directory of Open Access Journals (DOAJ). He also used the List of Publishers² which enumerates publishers which adopt unprofessional practices, also called “predatory”, according to Jeffery Beall, an information scientist at the University of Colorado, USA, the curator of the list. By cross-checking both journal lists in the subject areas of chemistry, biology and medicine, and eliminating those not published in English, the final list of targets came to 304 journals.
In order to create hundreds of different versions of the same “scientific” paper, the author began with the following premise : Molecule X from lichen species Y inhibits the growth of cancer cell Z. To create the papers and substitute for those variables, a database of molecules, lichens and cancer cell lines was created along with a computer program which allowed hundreds of “original ” papers to be created from a combination of these variables. The names of the fictitious authors were affiliated with equally fictitious African institutions using a program that allowed the permutation of African first and last names or the bringing together of prefixes and suffixes to generate the names of institutions.
The methodology section of the papers which had been generated in this way contained gross errors. The same held true for the analysis of the results from which it would be impossible to reach the conclusion that molecule X is a de facto inhibitor of cancer cell growth. More seriously, however, is the final conclusion of the paper which states that “molecule X is a promising new drug against the treatment of cancer”. This conclusion was based upon a small number of experiments which were at the very least inconclusive, and as an apparent advocacy of bypassing clinical trials. The final sting in the fraud was making the native English of the author more flawed and this was accomplished by translating the work into French using Google Translate and then translating the result back into English.
The Results
Between January and August of 2013, the author submitted works at the rate of about ten per week : one paper to each journal of each publisher. At first, the author chose journals which did not levy an article processing charge; in the second phase he chose those which only charged a fee if the paper was accepted; lastly, he chose those journals which required a fee to be paid up front for the submission of the paper. If a paper was rejected, then that was the end of the process for that journal. If a journal sent review comments and asked for changes, the author would send some photos of lichens, extra details on methodology, or better formatting without, however, changing any of the fatal scientific flaws in the paper. If the paper was accepted, the journal in question would receive a form e-mail in which they would be informed that the authors had unfortunately detected a flaw in the experiments which invalidated the results, and they were therefore asking for the submitted paper to be withdrawn.
The author concluded that many journals simply exist to rake in money from authors in the form of an “Open Access article processing charge” or to allegedly defray the costs associated with editing and reviewing an article. Many journals which are entitled “American Journal X” are in reality operating their publishing houses in China, India or Pakistan, and their editors are writing e-mails to authors in somewhat shaky English.
The Analysis
Since the date of publication of the article in Science, 157 journals had accepted the paper and 98 had rejected it. Of the remaining 49 journals, 29 seemed to have been abandoned by their publishers and were not responding to e-mails. They also didn’t have valid addresses in the places in which they stated they were operating. Editors from 20 other journals had informed the fictitious authors that their papers were still under review, and they were also excluded from this analysis. Acceptance took 40 days on average compared to 24 days to elicit a rejection.
Of the 255 papers (157 accepted + 98 rejected) that underwent the complete editorial process, approximately 60% apparently had not passed through any type of peer review process. This number is good news as far as rejections are concerned, but for acceptances it means that someone judged them to be suitable for publication without actually reading them.
The type of review requested by the 106 journals which made such a request focused basically on the formatting, layout and the wording of the language used. Of these, 70% ultimately accepted the papers. Only 36 of the 304 submissions generated review comments which took account of serious methodological flaws in the papers. Even so, 16 of these papers were accepted despite the superficial revisions which were carried out by the authors at the request of the reviewers.
The Beall list shows that it is fairly accurate in spotting publishers with poor quality control, or “predators”. For the publishers that completed the review process, 82% accepted the papers. The surprising result is for publishers in the DOAJ: of those publishers who completed the review process, 45% accepted the bogus article.
Renowned publishers such as Elsevier, which publishes the Open Access journal Drug Intervention Today, also fell into the trap, since it accepted Bohannon’s article for publication. Its Vice-Presidents, however, stated that this journal is not actually owned by Elsevier and that they simply published it for someone else, having no actual role in the editorial process. Other well-known publishers such as Wolters Kluwer, which publishes the Journal of Natural Pharmaceuticals which accepted the bogus paper for publication, made a public retraction and closed the journal down.
In the meantime, the author (Bohannon) has conferred with renowned scientists who care deeply about Open Access. They agree that the Open Access model per se should not be held responsible for the poor quality of the works targeted by Science. “If I had targeted traditional, subscription-based journals, I strongly suspect you would get the same result” says David Roos of the University of Pennsylvania. Everyone agrees, however, that scientific journals should honor their obligations to exercise strict quality control over what they publish.
Comments on the article made in the scientific and mainstream media
Peter Suber’s Google Plus
https://plus.google.com/u/0/109377556796183035206/posts/CRHeCAtQqGq
According to Peter Suber, one of the most committed and respected proponents of Open Access since its inception in the early 2000’s, Bohannon showed flaws in some OA journals, but these do not represent a significant sample of the OA category, nor are they worse than journals that are not OA. Nor is the problem of low-quality peer review restricted to OA journals, or that they are more dishonest.
In Suber’s opinion, the Science article is constructive in exposing bad OA journals and in alerting authors not to submit their papers to them. On the other hand, it has a detrimental effect on the general perception of OA, leading many to believe that all of these journals are of poor quality or dishonest. In fact the article seems to suggest that the majority of them charge an article processing fee which is not true because according to DOAJ the majority of them do not charge any fees at all. Bohannon, states Suber, is not responsible for the erroneous assessment that all OA journals are of inferior quality, but the way in which he draws conclusions in his article corroborates this false assessment.
Open Access Scholarly Publishers Association (OASPA)
http://oaspa.org/response-to-the-recent-article-in-science/
In its blog, OASPA published a public declaration regarding the Bohannon article where it points out the more serious flaws of the article and lists facts that seek to restore the credibility of OA, whose reputation was dented by the unethical practices of some publishers but not of most.
Like other bloggers and journalists, the OASPA points out the biggest limitation of the experiment, which is the lack of a control group and the non-randomized selection of journals to which the bogus paper was sent. The Association reaffirms its commitment to the ethical conduct of publishers affiliated to it and states that non-compliance by them could lead to their expulsion. However, the Association makes clear the most important message from Bohannon’s article, which is to identify, together with the academic community, new and efficient ways of picking out trustworthy journals and publishers, independent of their business model.
Retraction Watch
The authors of this blog (Adam Markus and Ivan Oransky) criticize Bohannon’s “experiment” for the lack of a control group which in this case would have been represented by subscription journals that are traditionally known for the careful peer review process that they undertake. To this criticism Bohannon counters by saying that such journals can take many months or even years to review a paper and, that in light of this, it would be difficult to obtain a significant sample in a time period comparable to that of OA journals. The authors even mention the non-OA Elsevier journal Applied Mathematics Letters which published a series of articles that did not make any sense mathematically speaking. The e-mail of the author of that series (ohm@budweiser.com) was also provocative in nature but this fact went unnoticed.
Inside Higher Education
Carl Straumsheim analyzes the experiment published in Science with a critical eye and puts himself in the author’s shoes. He cites examples of peer review processes under study in the field of the Humanities and concludes that there is not just one single peer review model. Many fledgling journals find themselves pressured to undertake a careful evaluation and still respond to the authors in a timely manner. Peer reviewers that undertake good reviews are difficult to recruit, given that their painstaking work takes time, is unpaid and does not receive the appropriate recognition. The editors of the OA journal Journal of Digital Humanities defend the proposal for a process of open peer review where “publication, criticism and discussion could all take place in the same space”. Yet, other publishers believe that more experience is needed to prove its validity. Ultimately, OA journal publishers are looking for an evaluation process that ensures both the diversity and quality of the articles.
Doubtful News
http://doubtfulnews.com/2013/10/the-slums-of-scientific-peer-review/
Sharon Hill stresses the fact that many OA journals have it as their objective to levy article processing charges only on those authors that have their works accepted. The study in Science also demonstrated that the most prestigious OA journal, PLoS ONE, passed the test, rejecting Bohannon’s paper. PLoS ONE can be considered a type of Open Access Science given its prestige and credibility. Hill admits, however, that the OA peer review process has taken a hard hit and needs to have its credibility restored but does not present options on how to do this.
Science Blogs – Pharyngula
http://scienceblogs.com/pharyngula/2013/10/04/stones-glass-houses-etc/
P.Z. Myers analyzes the work of Bohannon from the point of view that many journals that accept works generated by a computer are just simply a group of people, a computer and an Internet site with an attractive name, determined to collect publication fees. It is about a perfectly legal fraud, and the more articles accepted, the better.
It is unjust, however, to blame OA for the problem since 40% of the journals rejected the Bohannon article, just as PLoS ONE did. Myers also draws attention to the fact that there was no control group in Bohannon’s “experiment” which would have been the subscription journals. The author of the blog agrees that there are innumerable problems with regard to scholarly communication, however these do not relate to OA. In his opinion “ it’s an overproliferation of science journals, a too-frequent lack of rigor in review, and a science community that generates least-publishable-units by the machine-like application of routine protocols in boring experiments.”
The Guardian Higher Education Network
According to Curt Rice, the article in Science “…. doesn’t point to the real crisis – the meltdown of the peer review system”.
In general, the peer review process is taken very seriously by researchers. However, they are under pressure to publish because their career paths depend on it and works of poor quality slip through and are still published.
Rice cites the case of the researcher Dr. Staple at the University of Tilburg in Holland who published 55 articles based on fraudulent data in renowned journals, including Science.
According to Rice, Bohannon points to the crisis in science, which is not restricted to the universe of OA journals. What is in crisis is the system of peer review and – it is necessary to emphasize this – it is OA journals that are leading the development of new approaches to this important stage in the editorial process.
Rice also highlights that 65% of the journals in the DOAJ do not charge any article processing fees whatsoever, since they are maintained by universities or academic societies. He therefore recognizes that charging fees may favor corruption.
Like other authors, he also defends the open peer review project as an alternative, utilizing the potential of the Internet as the platform for scholarly communication today.
Scholarly Open Access
http://scholarlyoa.com/2013/10/03/science/#more-2386
The blog is authored by Jeffrey Beall, the author of List of Publishers , a list of publishers that have unethical practices which was used by Bohannon in his article in Science.
Beall wrote his blog post on the day before the date the article in Science was published, relating details of the work. He notes that despite Bohannon having withdrawn his work when it had been accepted by a journal claiming to have detected errors in the experiments, four of them published the article. The publishers of the four journals are in Beall’s List of Publishers, which gives this list credibility. Beall states “I’ve known for a long time that predatory publishers are corrupting open access, that they are accepting papers unworthy of bearing the imprimatur of science, and that they are hurting researchers, science, and science communication. I am delighted that the research reported in the Science article confirms this.”
Real Clear Science
http://www.realclearscience.com/blog/2013/10/spoof-paper-accepted-to-157-open-access-journals.html
“Open access. “Publishing for the people!” It’s a good idea, right? Yes! And no… sort of.”
Ross Pomeroy points out that the OA model in which the author pays to have the article made available freely to all is responsible for the low level of selectivity of the journals that adopt this model.
While Nature rejects 90% of papers submitted, the Swiss Open Access publisher Frontiers rejects just 20%. There is no reason to suppose that a journal published by Frontiers, considered one of the best OA publishers, would have fallen into the trap devised by Bohannon. In fact, the journal Frontiers in Pharmacology of Anti-Cancer Drugs rejected the paper.
Pomeroy also highlights the fact that PLoS ONE had rejected the paper, reinforcing its reputation for conducting a careful process of peer review. The same occurred with the publisher Hindawi based in Egypt which publishes Chemotherapy Research and Practice.
The blog’s author considers OA a worthwhile and valuable idea. However, says Pomeroy “reviewers and editors of open access journals must hold themselves to the onerous standards demanded by rigorous, proper science and not fall for the juicy temptations of easy money.”
Language Log
http://languagelog.ldc.upenn.edu/nll/?p=7584
Mark Lieberman, through the analysis of Bohannon’s article in Science and the comments of Curt Rice³ in The Guardian, reminds us that Science is the epitome of a subscription journal that, like other subscription journals, is suffering from the moral and political assault of the Open Access movement through recent mandates in the United States and in various countries in Europe which state that the results of publicly funded research should be made available in Open Access. It also must be considered that the cost of the publishing process of OA journals is not nil, and one of the ways to cover this cost is by charging article processing fees to authors. This model, for sure, is conducive to fraud and extortion which must be curbed, but this does not invalidate, per se, the OA publishing model.
Subscription journals and their publishers, it should be noted, are also motivated by high profits as evidenced by Elsevier and other publishers who earn large profits on the unpaid work of the academic editors, academics, authors and reviewers.
Lieberman agrees with Rice that the problem resides in the antiquated system of peer review in place today and ends by concluding that: (1) today’s system of peer review does not prevent the publication of works of bad quality, even in renowned journals; (2) its lengthy delays slows down innovation and development in science; (3) the proposed evaluation after publication may become the standard for publicly exposing the comments of a reviewer in a Web-based platform; (4) the model of open peer review can prevent rivalries, retaliation and biases that occur covered up by anonymity. However, Lieberman points out that scientists are conservative by nature, and a long time may be needed before changes are implemented.
External links
Blog Michel Eisen – http://www.michaeleisen.org/blog/
DOAJ – http://www.doaj.org/
Notes
¹ Science
Referências
BOHANNON, J. Who’s Afraid of Peer Review? Science, Vol. 342, nº 6154, pp. 60-65. Available from: doi: 10.1126/science.342.6154.60.
WOLFE-SIMON, F. et al. A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus. Science. 2010, Vol. 332, pp. 1163–1166 Available from: <http://www.sciencemag.org/content/332/6034/1163>.
About Lilian Nassi-Calò
Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.
Translated from the original in Portuguese by Nicholas Cop Consulting.
Related Posts:
Como citar este post [ISO 690/2010]:
Pingback: Comunicação científica – O início « Mario A. Lira Junior
Pingback: Open Access in Latin America free of predatory journals | Portal El Bohio
Pingback: Logic, reason and pseudo-science – Answers In Reason
Pingback: Logic, reason and pseudscientific nonsense - Welcome to Atheists Against Pseudoscientific Nonsense
Read the comment in Spanish, by Yazmín Alejandra Quintero Hernández:
http://blog.scielo.org/es/2013/11/05/polemico-articulo-en-science-expone-debilidades-en-la-revision-por-pares-en-una-serie-de-revistas-de-acceso-abierto/#comment-41043
Strange thing (scientific corruption ??) seen, read below:
Several Elsevier journals Editor-board favoring (corruption ??) publishing
own-papers-from-known-people style (questionable review, superfast acceptance,
inbreeding in journal).
One example:
Composite structures journal, ISSN: 0263-8223, Elsevier.
Editorial members fill-up own papers in own journal in multiple:
SCOPUS author ID: 7405608176
https://www.scopus.com/authid/detail.uri?authorId=7405608176
SCOPUS author ID: 7201745807
https://www.scopus.com/authid/detail.uri?authorId=7201745807
SCOPUS author ID: 7406669984
https://www.scopus.com/authid/detail.uri?authorId=7406669984
SCOPUS author ID: 7006373871
https://www.scopus.com/authid/detail.uri?authorId=7006373871
Paper accepted in 2 or 3 days of (doubtful) review:
http://www.sciencedirect.com/science/article/pii/S0263822317323802
http://www.sciencedirect.com/science/article/pii/S0263822317333202
Good good.
Indeed, AM, this is how you get an h index of 78 or 31! Their peer reviewers might have super powers to work that fast – and flawless. Thanks for your input.
Report by AM above is valid and it is corruption going on. Need investigation and retraction.