[[ This document is a synthesis of guidelines from past ATC as well as other USENIX systems conferences ]]
There are two rounds of reviews, R1 and R2. The first round of reviews will be due in early March. We will then eliminate the low-ranked papers and assign a second, smaller round of reviews. At the end of each round we will use the online discussion/comments HotCRP feature for specific papers that warrant discussion. Thus, we are likely to reach out to you shortly after each round of reviewing.
LONG and SHORT Papers
ATC ‘18 accepts long (11 pages) and short (5 pages) papers.
An ideal short paper would express an idea that doesn’t require 11 pages to describe or evaluate. The ideas should be well formed and complete just like those in a Full paper. A Short paper is NOT a HotX-style paper (e.g. HotOS, HotCloud, HotStoarage). That is, a short paper should not just contain an interesting idea without any implementation and evaluation. When reviewing Short papers, our recommendation is to hold them to the same standard as Long papers in terms of clarity of presentation and evaluation. However, the scope of the work as well as the breadth of the evaluation is smaller compared to the Full paper and this should be taken into consideration when reviewing a short paper.
We will not convert short paper submissions into full papers or vice versa. Instead, we ask you to evaluate the paper as submitted, not what it should or could have been had it been submitted as a different-length paper.
REVIEW GUIDELINES
The review form on HotCRP includes explanations for each field, so we’ll just offer some broader guidance here.
There are 2 ranking scores for each paper. The most important one, of course, is the “Overall Merit” score (scale 1..5), which is your recommendation for whether we should accept or reject the paper. In assigning these rankings, we not only want to identify the best research, but also come up with an exciting program that will stimulate discussion and excite people to attend.
There is no quota for how you distribute ratings across the papers. Rate each paper based on its own merits and its suitability for ATC. But FYI, we try to accept 20% (or a little bit more) of the papers. You can use this number to calibrate your ratings. A good paper is a good paper, so please be positive.
We expect that papers with no accept ratings (weak, regular, or strong) after the first round will be rejected and not reviewed further. However, we will give PC members a chance to “resurrect” papers about which they feel strongly positive.
“Reviewer Expertise” (scale 1..4) is the second most important rating field. We will be loathe to make decisions about any paper that does not have at least some knowledgeable or expert reviewers. We will be paying close attention to these scores to decide where we need to solicit additional or external reviews. To help us with this, please let us know via email as early as you encounter any such paper for which you have particularly low expertise; we may use this information to promptly assign additional reviews if appropriate.
Finally, the last important part of a review is the free form commentary which is extremely valuable. Please do not submit a review that includes only numeric scores only and no text. Often, your choice of words will be instrumental in clarifying to everyone your feelings about a paper. Please do use the free form sections to explain the reasons for your ratings—especially for the accept/reject recommendation. Note that your audience in writing this portion of the review is both the program committee and the authors. While it is important to give your most accurate assessment of a paper, please keep in mind that a criticism can be often phrased in a constructive manner.
EXTERNAL REVIEWS
Please do all of the reviews yourself. We will primarily use external reviewers for papers where we don’t have enough expertise within the committee or where an external reviewer may offer a unique and valuable perspective on a paper. If you are reviewing a paper and believe an external review would be helpful, please let us know first—either by e-mail or by suggesting an external reviewer in HotCRP. All external review requests will come from one of us, so we can check for conflicts before assigning the review.
WRITING GOOD REVIEWS
The authors of the submissions are likely to have put in tremendous effort and value the feedback from the ATC reviewers. Thus, please do bear the following guidance in mind as you write review feedback.
Please provide positive and constructive feedback even if you dislike the paper or disagree with specific technical content.
Please try to critique the paper as is; while suggestions for improvements are certainly welcome, simply listing things that the authors could have done instead is not helpful for determining the merit of the current submission.
Reviews need to strike the right balance between brevity and length; in general, brief reviews are not particularly useful for the authors to make informed improvements. Please try to provide detailed feedback especially when being critical.
If you don’t find the paper, or parts of it, novel, please try to provide appropriate references to the prior work(s).
A few additional sources of advice on writing reviews:
https://people.inf.ethz.ch/troscoe/pubs/review-writing.pdf by Timothy Roscoe
https://people.eecs.berkeley.edu/~fox/paper_writing.html#rev by John Ousterhout
https://www2.eecs.berkeley.edu/Pubs/TechRpts/1989/CSD-89-511.pdf by Alan Jay Smith.
As you review papers, please don’t hesitate to contact us via atc18chairs@usenix.org with any questions or issues that arise.