The points contained here are a summary of those from https://medium.com/@deviparikh/how-we-write-rebuttals-dc84742fece1
Overview
- For the reviewers:
- clarify doubts
- answer questions
- correct misunderstandings / push back on mischaracterizations
- make a good-faith effort to incorporate feedback
- For the AC:
- convince them that you have made a good-faith effort
- present a representative summary of the reviews
- help them understand if the reviewer concerns were addressed
- call out bad-faith reviewing
- ultimately, help them make a decision.
(In our experience, most new members of the research community focus on (1), but ignore (2))
Key question
Would a neutral third-party be able to tell if the reviewer concerns were addressed purely based on your rebuttal?
Process
Itemize reviewer comments & brain-dump possible responses
They use this spreadsheet to organize individual comments/questions/concerns. Dump your thoughts here in rough text without regard for style or length. Being convincing and concise is a subtractive process.
Write a draft rebuttal
Turn your consensus in the sheet into concrete responses in a rebuttal draft. Write concisely but don’t worry about space. Cover every point and trim / prioritize them later.
Review and revise
Reread the initial reviews and your sheet to make sure everything has been
addressed. Prioritize major concerns and start working towards meeting space limitations.
Tips
Start positive
Provide a summary of the reviews, highlighting positive things that reviewers said about your work. Rebuttals focus mostly on responding to perceived negatives, don’t let RACs forget about the strengths along the way.
Order matters
Start with the biggest concerns that you have good answers for and work your way to less clear-cut responses and minor points.
Use the reviewers' own words, then respond directly
Quote the core of the reviewer’s question or concern concisely and completely. Then before saying anything else, respond to it directly. And then give details, describe context, or explain your position. E.g.
- “Are these averaged across multiple runs?”: “Yes, we averaged across 5 random seeds.”
- “So overall, the proposed approach needs more human annotations than the baseline.”: “Not quite. While the first few iterations of our approach...
- “Did you evaluate on realistic environments?”: “We disagree with the question’s premise. While these environments are simulated, they are highly photorealistic.”
Â
Be conversational
Notice the conversational nature of the example responses above. It
makes it easier for RACs to follow, and the responses are less likely to
be perceived as being combative.
Respond to the intent of the questions
Don’t feel trapped to only discuss the quoted concern — also address the intent of the comment. For example, “Why didn’t you evaluate on GLORP3?” may generally be calling your experiments into question. Answer, but then point out that you’ve already evaluated on X,Y, and Z which should be sufficient! Note that it is useful for other RACs to be reminded of
your extensive experimental evaluation. A first glance at a reviewer comment suggesting otherwise could leave a false impression.
Don’t be afraid of emphasis
“Row 2 in Table 4 shows exactly that.” “We do not need a human-in-the-loop at test time.” Notice that many of the responses above are not just direct, but also have emphasis (in tone if not formatting of text).
Feel free to set the stage
Keep things self-contained
Assume RACs don’t remember much about your paper and that they likely won’t read it again in detail. Re-introduce any acronyms, remind them of relevant details of an experimental setup.
Get credit for details you already included
If something a reviewer asked for was already in the paper, say so. Give them line/Table/Figure numbers, and then restate it in the rebuttal. The references back to the main paper are to establish credibility with all RACs that the paper was not lacking important details. (They are not necessarily to have RACs go back and look at the paper.)
Consolidate common concerns
Save space by responding to multiple reviewers at once if they share related concerns.
Color-code reviewers
Notice above the trick to color-code reviewers. Make it as easy as possible for reviewers to spot responses that are relevant to them—even when things are merged or not in reviewer order.
Stats speak louder than words.
Rather than argue with RACs, give them data/stats to back your claim up. These can be statistics/analysis of your data or results. Or the results of additional experiments you run to respond to their concern (if allowed by the venue). Every time you find yourself having a different opinion than the reviewer, ask if you can establish that with data. You can always provide the intuitive arguments after you’ve settled the issue with data.
Don’t promise, do
Instead of saying “We will discuss Singh et al. in the paper.”, provide a discussion in the rebuttal. Instead of saying “We will explain what D_{RT} stands for in the paper”, explain what it stands for in the rebuttal. And then also add that you will add it to the paper. It makes it significantly easier for RACs to trust that you will make the promised changes.
Be receptive and reasonable
Most RACs will appreciate it. Plus, it is just the better thing to do — these are your colleagues! :)
Likewise, instead of arguing why a reviewer’s suggestion might not work, try it and see if it works. Maybe it will!
Be transparent
Reviewers hinted at an additional experiment but the venue doesn’t allow it? Say so. They asked about intuitions about a trend but you don’t have any? Say that you’ve thought about it but don’t have any good ideas, and will continue to investigate it. Don’t have enough GPUs to run the experiment they asked for? Say so.
Shine a spotlight on reviewers acting in bad-faith
In some circumstances, a reviewer may not be adhering to reviewing best practices or may not have taken the reviewing role seriously. It can be
important to make sure the other RACs realize this and appropriately discount their review. Pointing out unreasonable or unsubstantiated comments and referencing other reviewers that disagree can help. This can also include confidential comments to the AC (where applicable).
Acknowledge reviewer efforts
On the other hand, if a reviewer goes above and beyond to be constructive, thank them for it. Typo list? Thank you. Pointers to relevant work? Thank you. Detailed musings about future work? Thank you. Add at least a short blurb acknowledging these things!
Don’t forget the humans on the other end
Keep in mind that this is not just a scientific but a sociopolitical interaction with other humans :) So decide whether you’d like to be argumentative and risk your reviewer taking a strong stand against the paper, or if you’d like to work towards a common ground. Finding points where you do agree with the reviewer and acknowledging them can help with the latter.