One of the first assignments I got when starting my PhD(4 weeks ago) was to review some papers the lab had done, that I would possibly be doing extending work on. As I have never reviewed papers before I thought a good idea would be to go at it like I was reviewing the paper for a conference, journal etc. Googling and reading advice from a number of sources I came up with a check point list and some general advice. I have rewritten most of the advice I gleamed from the authors of the sources, however think of the notes as more of a copy & paste work than any original thoughts.

“Before submitting a finished report, a wise referee asks – Would I be embarrassed if this were to appear in print with my name on it?”

Also interesting and a fun read is the article by G. Cormode, How NOT to review a paper – The tools and techniques of the adversarial reviewer.

Question to ask for each part

Most papers consists of the following parts: title, abstract, introduction and/or related work, methodology, results, discussion and/or conclusion, references, and tables & figures. So what questions should you check of when reading a paper for the individual parts? This could also be used as a checklist before sending your papers off as to minimize the collateral damage.

Title
– Does the title describe the actual work or is it just fancy wording? Can you from the title categorize what it is about?

Abstract
– Does it inform clearly as to what was done, what the major findings and results were?

Introduction and/or Related Work
– Is there a clear problem formulation and does it correctly describe the aims of the author?
– Does it motivate why the problem is interesting?
– Does it put the work in the proper context of previous work and related work, i.e. has the authors done the background reading properly?
– Does it explain what previous work it builds on, or challenges, extends etc.?

Methodology
– Does it account for the data collection, and how it is being used?
– Is the data and method used proper for answering the problem formulation?
– Is there enough information for replicating the research?
– Is the methodology clearly explained and ordered in a logical way? (Refers to the previous point.)
– If new/unknown methods are introduced are they accounted for detail?

Results
– Are the results good enough to warrant a paper?
– If not, does the results give new information into the field/problem being researched?

Discussion and/or Conclusion
– Does it summarize the major findings and results?
– If the results are completely different from what was anticipated is it properly explained, concluded and discussed what they mean?
– Does it give ideas of it could be extended or is it a dead end?

Tables & Figures
– Relevant and important?
– Consistent?
– Readable?
– Does the caption explain the graphics in an understandable manner?
– Does the diagrams describe the data accurately?

References
– Are all references and citations formatted in a consistent and clear way?
– Are all references fully informational i.e. author, title, date published etc.?

Ideas on originality
– Is this sufficiently novel and interesting to warrant publication?
– Does fit well into to the canon of knowledge or is it an outlier?
– Is the research question or the problem solved novel enough?
– Does it reach the journal’s/conference’s standards?
– Does it fall in the top 25% of papers in this field?
– Will it interest people who attend this conference / read this publication?

Categorizing novelty
One of the sources lists the following numbering to bin the paper, which should help in the categorization work.
(1) Major results; very significant (fewer than 1 percent of all papers).
(2) Good, solid, interesting work; a definite contribution (fewer than 10 percent).
(3) Minor, but positive, contribution to knowledge (perhaps 10-30 percent).
(4) Elegant and technically correct but useless. This category includes sophisticated analyses of flying pigs.
(5) Neither elegant nor useful, but not actually wrong.
(6) Wrong and misleading.
(7) So badly written that technical evaluation is impossible

General advice when giving critique

There seems to be sort of consensus on the art of giving good constructive critique, and when a paper is good enough for publication. I summarized the most important parts below ( And again all ideas are basically rewrites or citations of ideas of the sources I am not taking credit for anything).

– “Your review should not be about what should have been done; rather it should be a critique of what they authors actually did. If you feel the authors should have done something else, accept the paper and discuss it with them at the conference.”

– “You should compare the paper with the average paper in that specific journal or conference, not with the best or worst. Of course, in some cases the average is too low and needs to be raised by critical refereeing.”

– “Refereeing a paper can require considerable time and effort; don’t waste that effort on a detailed critique of a badly flawed paper that can never be made publishable. Finding one or more fatal and uncorrectable flaws excuses the referee from checking all subsequent details.”

– “Small results which are surprising and might spark new research should be published; papers which are mostly repetitions of other papers should not; papers which have good ideas badly expressed should not be published but the authors should be encouraged to rewrite them in a better, more comprehensible fashion.”

– “You should summarize the point of the paper in one to five sentences, both for the editor’s use and to ensure that you actually understand the paper.”

– “If your recommendation is favorable, you must list both necessary and suggested changes. If your recommendation is negative, but you think the paper can be salvaged and either submitted elsewhere or resubmitted, then you should provide a similar (but perhaps less detailed) list. Suggestions for alternate places to publish are always welcome.”

– “Review to accept papers.”

– “If there is something so critical that it MUST be included, suggest something to remove/reduce so that the authors can kept to the page limits.”

– “Likewise, don’t demand any additional work that can’t be done in the time between acceptance notification and the final submission deadline. (As a more general rule, never reject a paper for something that can be fixed in 5 minutes.)”

– “In general, write your entire review in a tone of having accepted a paper, even when you’re not recommending acceptance. This will help change what you subconsciously write as condemning criticism to helpful comments.”

– “Consider giving a general comment on the article on e.g. novelty and overall impression of the data and manuscript preparation.”

– “List major comments. Number them for clarity. Major comments are comments, questions and/or suggestions that are in your view essential points that need to be appropriately addressed for the manuscript to become acceptable for publication.”

– “List minor comments such as typographic errors or suggestions for additional non-essential data to be included.”

– “Be kind. Be fair. Be concise. Be ‘action-able’.”

Links / Sources
(All citations and information comes from these pages, in no particular order. I did note have time or energy to properly do a reference for every citation.)
www.paperpub.com.cn/admin/upload/file/2009421114758398.pdf
www.bcl.hamilton.ie/~barak/how-to-review.html
www.cs.uiuc.edu/homes/caesar/classes/CS598.F08/readings/reviewing.html
www.cgl.uwaterloo.ca/~smann/Research/review-conference.txt
www.nieronline.org/index.php?title=How_to_review_a_scientific_paper%3F
www.cl.cam.ac.uk/~ey204/teaching/ACS/R202/aid/stevens.pdf
users.ecs.soton.ac.uk/hcd/reviewing.html
eetd.lbl.gov/EA/Buildings/ALAN/PUBLICATIONS/how.to.review.html
ccr.sigcomm.org/online/files/p83-keshavA.pdf
www.cs.washington.edu/homes/mernst/advice/review-antipatterns-devanbu.txt