On the value of critical evaluation in science

(First of two parts)

The productivity and the quality of work performed by individual scientists, research groups, departments, universities, countries and continents is evaluated by the number of papers published in international peer-reviewed journals, by their citation rates (the number of times these papers are cited by others over time), and through peer review. Evaluation processes, though imperfect, make use of such quantitative measures as well as the qualitative assessments provided by experts. Evaluations might include a site visit, called an “external review,” wherein scientists from other institutions visit a laboratory or institute and work as a team to conduct an in-depth analysis. A report is written and external reviewers provide recommendations that may range from increased funding and support to the disestablishment of units that are unproductive or dysfunctional. In certain countries, this process is considered so important that external reviews are conducted at regular intervals and costs are borne by the institutions whose academic or research units are under scrutiny.

The critical evaluation of the quality of scientific work is widely accepted as being both desirable and beneficial. Science cannot be done well and would not advance if scientists were not skeptical and if they believe everything they read or hear. For example, to properly train a bright graduate student from China, a British professor instructed her to disagree with him on a regular basis and to question whatever she read in scientific journals. Such healthy skepticism is part of the scientific culture; scientists expect seminar speakers to support their claims with data and to discuss how hypotheses, whether implicit or explicit, have been subjected to empirical test. How could a series of unsupported assertions be considered a scientific seminar? Several years ago, when a prominent scientist finished his seminar, my Ph.D. supervisor stood up and challenged his claims, starting with the words “There are at least five reasons why you are wrong.” Such exchanges among scientists are not unusual in many countries. On the other hand, the absence of even more polite versions of this in certain cultures can make way for the acceptance of nonsense.

The importance of critical evaluation also explains why acceptance for publication in international, peer-reviewed journals serves as a test of the quality of scientific work. But because, sometimes, erroneous papers do get accepted, good scientists are skeptical even of published work. Letters to the editor that challenge published papers are common. The discovery of errors or fabricated data by other scientists leads to the publication of either embarrassing corrections or potentially career-ending retractions by the original authors. It is not uncommon for experiments to be replicated by others to verify results or to take the work in a new direction. In seminar classes, graduate students are trained not just to look for flaws in published papers; they are trained to evaluate quality — what these papers add to what we already know and whether what is added is trivial or important.

(To be concluded)

* * *

Raul Kamantigue Suarez is a professor in the Department of Ecology, Evolution and Marine Biology of the University of California, Santa Barbara, California and an editor of the Journal of Experimental Biology, Cambridge, UK. E-mail him at [email protected].