Guide to Reviewing Papers and Notes

This page provides guidelines for reviewers responsible for assessing submissions to CHI 2016.

Key points:

• Your primary criterion for judging a paper is: Does this submission provide a strong contribution to the field of HCI? Remember that there are many ways a paper can make a contribution to HCI, and you should review the paper appropriately. See “Contributions to CHI” for details.
• Reviewers rate each Paper or Note using a 5-point ranking scale; your written appraisal must support your numeric ranking.
• A high-quality review is typically about a page of written text; very short reviews are frustrating for authors and hurt the review process.  Always put yourself in the author’s position: what level of detailed feedback would you like to see for your own work?

Contributions

The primary criterion for evaluation of all Papers and Notes is the submission’s contribution to HCI. In all cases, a CHI Paper or Note must break new ground and make an original research contribution. However, it is important to recognize that there are many ways for which a paper can make a contribution to HCI, and you should review the paper appropriately. Please see “Contributions to CHI” for a list of some of the types of contributions a paper can make to HCI, and the associated criteria that you can use to assess this type of contribution.

Papers and Notes

Papers and Notes are both reviewed within the same rigorous review process and at the highest level are judged by very similar criteria (i.e., does this Paper or Note provide a strong contribution to the field of HCI?). However, it is important as a reviewer to realize that the type of content that is appropriate for each is somewhat different. In particular, Notes present brief and focused research contributions that are noteworthy, but may not be as comprehensive or provide the same depth of results as a Paper. For details about these differences, see Papers Versus Notes: What’s the Difference?

Prior Publication

Content appearing at CHI should be new and ground-breaking. Therefore, material that has been previously published in widely disseminated publications should not be republished unless the work has been significantly revised. Guidelines for determining “significance” of a revision are stated in the ACM Policy on Pre-Publication Evaluation and the ACM Policy on Prior Publication and Simultaneous Submissions. Roughly, a significant revision would contain more than 25% new content material (i.e., material that offers new insights, new results, etc.) and significantly amplify or clarify the original material. These are subjective measures left to the interpretation and judgment of the reviewers and committee members – authors are advised to revise well beyond the policy guidelines.

An exception is for work that has previously been presented or published in a language other than English. Such work may be translated and published in English at CHI. The original author should typically also be the author (or co-author) of the English translation, and it should be made clear in your submission’s abstract that this is a translation.

Note that this policy on prior publication refers only to re-publication of one’s own work; this does not preclude publication of work that replicates other researchers’ work.  Novelty is highly valued at CHI, but constructive replication can also be a significant contribution to human-computer interaction, and a new interpretation or evaluation of previously-published ideas can make a good CHI paper.  Also note that a CHI paper should not be rejected on the grounds that it overlaps with work developed independently that was published after the CHI submission was made, during the review period.  In other words, work that an author couldn’t have known about shouldn’t count against him or her.

Subcommittees

To improve the reviewing process, the CHI program committee is divided into nine subcommittees.  Each sub-committee is responsible for a topic area within HCI (see Selecting a Subcommittee for details). Each subcommittee will consist of between two and six Subcommittee Chairs (SCs) and several Associate Chairs (ACs) who are knowledgeable in these topics. As specialists in this topic area, ACs’ primary responsibility it to find good reviewers (such as you) for each submission.

However, as a reviewer, you should not judge the paper by how well it fits the subcommittee theme(s). Many papers will not cleanly fit into a particular subcommittee for a variety of reasons, and we do not want to penalize authors for this. Remember, the subcommittee organization is there only to try to improve reviewer matches and to better handle the volume of submissions. If you have a paper that does not fit the subcommittee theme, evaluate it as best you can with respect to its own quality. Any topic is valid, as long as it fits within the interests of a reasonable fraction of the overall CHI audience. The primary criterion for review is the submission’s contribution to HCI.

How You Fit Into the Review Process

For more information about the overall CHI review process, see “CHI 2016 Papers and Notes Review Process“.

References

We highly recommend Ken Hinckley’s thoughtful piece on what excellent reviewing is.  If we had any way to enforce this, we would make it “required reading” for CHI reviewers and ACs:

http://mobilehci.acm.org/2015/download/ExcellenceInReviewsforHCICommunity.pdf

Even with great guidelines like these that we can all agree on, the debate about what makes a good CHI paper has been going on as long at the CHI conference has existed. If you are interested, the papers below touch upon this debate and contain references to additional papers that concern it.

• Greenberg, S. and Buxton, B. 2008. Usability evaluation considered harmful (some of the time). In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. CHI ’08. ACM, 111-120. DOI=http://doi.acm.org/10.1145/1357054.1357074
• Olsen, D. R. 2007. Evaluating user interface systems research. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology. UIST ’07. ACM, 251-258. DOI=http://doi.acm.org/10.1145/1294211.1294256
• Dourish, P. 2006. Implications for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’06, ACM, 541-550. DOI=http://doi.acm.org/10.1145/1124772.1124855
• Newman, W. 1994. A preliminary analysis of the products of HCI research, using pro forma abstracts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 – 28, 1994). ACM, New York, NY, 278-284. DOI=http://doi.acm.org/10.1145/191666.191766
• Daniel Reed and Ed H. Chi. 2012. Online privacy; replicating research results. Commun. ACM 55, 10 (October 2012), 8-9. DOI=http://doi.acm.org/10.1145/2347736.2347739