Does peer-review guarantee the quality of scientific papers?

The peer-review system is adopted by scientific journals in order to validate the content of articles submitted by authors for publication.

When one or more authors send a manuscript to a journal for publication, the editor (the person in charge of publishing it) sends it to two or more (presumed) experts to verify that the article is error-free and scientifically correct. This last point already lends itself to some criticism because it is not very clear what it means. Science hardly lends itself to “ordinary” or standardized evaluations, although, in fact, almost all published articles are rather trivial, in some ways (in the sense that articles often contain more or less relevant results, yet to obtain them it is sufficient to correctly apply a series of “rules”).

Usually the anonymous referees send their comments to the author and, in copy, to the editor. If they consider the content of the article substantially correct, they give the go-ahead for publication, often indicating some changes to be introduced or asking for clarifications. Otherwise the article is not published and the author receives a letter of comments explaining the reasons for such a decision.

Most researchers today believe that the system works and guarantees the quality of published material and, above all, think that the mechanism that allows the publication of scientific articles has always been this.

In fact, some people (including myself) openly criticise the system because it lends itself to abuse and does not guarantee the progress of scientific research at all. Some critics (including myself) think that it may even damage it.

First of all, it should be made clear that the peer review mechanism has not always existed and is a relatively recent invention. Many scientific journals are called Letters (Physical Review Letters, Physics Letters, etc.) because in the past scientists used to write letters to each other to make their research known. And it is clear that a letter is not subject to any review mechanism.

Not even when they began to publish in a journal, for the sake of convenience, did they imagine setting up such a mechanism. In fact, there was no need for it. Those who published in certain journals were to some extent certified to have a certain reputation for being a member of the academy or industry.

The peer-review system was recently created to select relevant articles from the growing number of articles coming to the editors of journals or as a form of help for editors who, given the wide range of knowledge required to judge the publishability of a manuscript, were not always able to decide independently.

As recalled in a recent post by Andre Spicer and Thomas Roulet , brought to my attention by Nathalie Lidgi-Guigui, Albert Einstein’s works were not subject to this practice. Probably, if they had been from the beginning, they would never have seen the light. In fact, Nathalie publishes an excerpt from the letter in which Einstein asks for the publication of one of her articles, without saying what it is about, and asks her colleagues to help her answer as a referee. This is the post.

Image for post

Almost everyone ridiculed the poor author, until Nathalie revealed who it was. Spicer and Roulet’s post remembers how Einstein, having heard that the editor had sent one of his manuscripts to a referee, not believing he had the skills to judge it, replied beaten:

We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorised you to show it to specialists before it is printed. I see no reason to address the — in any case erroneous — comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.

Wow! The fact is that not everyone is Einstein and most of us can’t afford to answer a journal editor this way. Science is made by a few great people who naturally can do their work thanks to the silent work of many “labourers” like me who, by taking measurements and publishing them, do a great service to the community and allow geniuses to use them creatively. Someone, however, consider him/herself a genius (not being such) and demands the publication of inconclusive, wrong or, worse, totally or partially constructed research. It is therefore necessary to put a stop to this practice and so began the process that today is universally considered the only possible and completely “natural”.

Unfortunately, this process has been grafted onto the so-called objective evaluation of research, which aims to measure the quality of a researcher by the number of publications and their citations. Incidentally, I note that science, in the meantime, has moved from an apparently rigorous definition of probability to one that defines the latter as the result of a subjective evaluation (which does not at all mean that elements of objectivity, which are often more solid than those extracted from an objective definition of probability, cannot be drawn from these evaluations). This has sparked a rush to publication for which today such a large number of articles are produced that only a very small part of them could materially be read by a researcher in each individual sub-sector. And this race has stiffened the position of referees who often take their role too seriously or completely misunderstand it, inserting elements of presumed objectivity into this race.

On average, the referee feels compelled to judge the content of an article, but he often does so on the basis of his own convictions without assessing whether or not the author’s arguments make sense, especially if the proposed research is innovative. It is clear that, especially at the beginning of the construction of a theory, not all the aspects can be clear even to the author him/herself. Many referees tend to reject the article if it does not present a fully coherent and complete view of all phenomena and aspects (or most known ones) that might be involved in principle. Can you imagine what would have happened to the article on the photoelectric effect in this case? Or to those on black body physics?

In general the referee also tends to push the author to write what he would like to write, in the way he would like to write it. Files, fixes, deletes and rewrites sentences, without limiting him/herself, as he/she should, to point out the possible need for linguistic revision where the author does not dominate the language in which he/she writes. He/she introduces in his/her judgment his/her own vision and personal interpretation of the data or models presented.

The apex is reached with the analysis of the bibliography. Now I will say something that will make many readers nervous: the bibliography in an article is almost completely useless. I was tempted to reply to Nathalie’s post that to me the author’s request to be exempted from the illustration of the bibliography did not seem so meaningless, although a little arrogant, especially in the way it was presented, but then I did not do it so as not to appear, as usual, “opponent”. Of the dozens of articles cited in a manuscript, less than 5% is often enough to make the article completely comprehensible and to allow those who wanted it to go deeper. The function of the bibliography in an article is this (apart from thanking, sincerely, the authors whose articles have been seriously helpful in the preparation of the manuscript). On the other hand, if you want your article to be published, you must include at least a whole page of bibliographical references to be cited everywhere, possibly in the introduction. You must show that you have read a lot and appreciate the work of many colleagues. Because after all, one of them, or a close colleague, could be a referee of your work. And if that one doesn’t find his/her article mentioned in yours, he/she will note it. It is important that the bibliography is long and exhaustive because, statistically, sooner or later it will be the referee’s turn to be cited with the same mechanism.

Not to mention articles in which we would like to illustrate our own opinion or a different interpretation or a rereading of already known phenomena, in order to start a debate in the community. Or even those in which you present an experience because you believe it could be useful to others who could implement it as it is or improve it. This kind of communication has almost no place in a scientific journal. I’m not saying it’s not correct, but the fact is that there are practically no journals dedicated to this and so it’s very difficult for an author to make his point of view on certain topics known or to gather the opinions of others to involve them in the discussion.

What’s the solution? I confess that I do not have one, but I would be content with the fact that the referees do their work honestly and without excess: when I judge an article as a reviewer, I try not to impose my vision of things (I can suggest it in the letter I send to the authors to consider it, but without expecting it to be accepted); I point out the need for a linguistic revision, if it is worth; I limit myself to checking calculations and consistency of symbols and notations; I do not intervene in the interpretation of the results, unless it is clearly in conflict with the data; I do not judge in advance the impact that the article may have on the community (this is actually a mandatory indication required by the editor, but as far as I am concerned it is never a criterion for rejecting an article). I believe that if an article is not contradictory and inconclusive, the community will judge it: if the content is copied or invented, someone will denounce it through the publication of another article. One can certainly not expect the referee to know all the literature related to the subject, so it cannot be the latter’s responsibility for the publication of a plagiarism or a scam.

Written by

Professor of Physics at Sapienza Università di Roma. Member of the CMS and PADME collaborations. Arduino advocate and phyphox ambassador.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store