A Brief Essay on Reviewing Scientific Papers

Brian Shoichet, UCSF

Established scientists, especially academics, are regularly asked to review papers. Doing so twice a month is common, and a rule of thumb is that for every paper you publish, you should review two. Recently several excellent resources have been developed for paper reviewing, particularly by the Fraser lab (https://fraserlab.com/peer_review/). Here I add to this guidance with a brief sketch.

The process begins when a journal editor asks you to review a manuscript. If the work is in your field, the review may take you four hours, more if you must consult the literature. The first questions are is it in your field, and can you squeeze it in? If both answers are yes, and if the journal is one that you publish in, or that you might want to publish in, you should accept the assignment.

Your review begins with a summary of the manuscript’s key findings. What was its motivation? Is it a question with which others have grappled; is it important? What were its key observations? This introductory paragraph may run to eight sentences, it is your summary of the paper, not so much your opinion of it. (Occasionally, manuscripts you receive have gaps so great that they prevent even a proper review. In this rare event, you can send the manuscript back suggesting that it be withdrawn from review until these gaps are addressed.)

Summarize your opinion of the study in the second paragraph. Is it scientifically sound? Is its impact high enough to justify publication in the journal? Many reviewers dislike judging impact (who are you to say whether it is Nature worthy?), and some have suggested separating impact and soundness into evaluations by two separate reviewers. Until the world is remade this will be impractical, but as a reviewer you can separate the two points in your thinking. Following your overall evaluation begin detailed points that the authors should address for publication, either in the current journal or elsewhere.

  1. Enumerating your points helps authors, editors, and you to track their responses.
  2. Identify the big issues, ignore the small stuff. If you mix the two your most important points can be lost in the chaff, and you may come off as churlish.
  3. Cite the figures or tables where the data you are critiquing are found, or the sections where the authors argue points with which you disagree.
  4. Identify the experiments or arguments that would address the key weaknesses and that would satisfy you on a second round of review.

In writing your critiques, imagine you are speaking to the authors, and put yourself in their shoes. Some reviews read as unkind or arrogant, and it’s natural for people to be sensitive about their work. Even if you find major flaws, say so in a professional and courteous way. While the senior author may have published many papers, this may be the lead author’s first or second ever. If you find the work thorough, but not impactful enough for the journal, commend the authors on its soundness, even while remaining clear that the work is better suited to a more specialized venue.

Reviews may end with a two-sentence summary of key strengths and weaknesses, points to address, and impact. If you have gotten into the weeds, this is a chance to revisit the key points for the authors to address in revision, and for the editor to evaluate in their decision.