Once the final project have been presented in the final class, you will be assigned one or two (see below) projects to review. You will personally create and submit a review for each project, engage in an anonymous discussion with other reviewers, and as a group will converge on a single final "meta-review" to be sent to the project authors. All of these activities will take place within the online EasyChair conference management system; you will receive email about this late in the term. You will not review your own project or that of anyone you have a conflict-of-interest with (see COIs below).
Note that the student reviews will have no influence on the grade you receive on your project; the main purpose here is to give report authors helpful and constructive feedback on their work.
Note also that your participation as a reviewer will form a
part of your final grade.
In addition to providing feedback to the report authors, taking part in the review process will give you some experience of how blind peer review works in scientific conferences. While you should be critical, please watch your what you say; as much as possible, keep your comments constructive and positive in tone.
The reviews will be "triple-blind" meaning that (a) the project authors will not know who reviewed their paper (and will not be able to see any of the preliminary reviews or the discussion comments), and (b) the reviewers will also not know the identities of the other reviewers they are having an online discussion with. Normally, triple blind reviewing would also mean (c) that the reviewers don't know who the authors are, but obviously that won't hold here.
I will act in the role of conference program chair, and I will be able to see everyone's identity, and provide any mediation that might be needed. Note that the first person to submit a review for a given report will be know within the EasyChair system as "Reviewer #1" or "R1", the second as "Reviewer #2" or "R2", etc.; this numbering is assigned per report, so Reviewer #1 for paper A will probably not be the same person as Reviewer #1 for paper B. In the online discussions, please refer to each other as "Reviewer #1" or "R1" etc.
Please do not break your anonymity inside or outside of the system.
The deadlines for each phase can be found on the course schedule page.
The project reports are due during week 12 and the authors will enter them into the EasyChair conference management system as a research paper. Once the presentations have been given in class, I will make the reports available for students for review, and I will assign some number of projects to each student. You will enter you reviews into the system; once you have done so, you will be able to see the other reviews that have been entered so far (and you will then start to receive email as new reviews and comments are added in). The initial reviews will have a due date after the last class but before the last day of classes (see the schedule page); but you are encouraged to submit your review as soon as you can. Once the initial review deadline has passed, you will hold an online discussion inside EasyChair (tho you can start it earlier if you like). You can add comments and also revise your review within.
For each project, I will assign one student as a lead reviewer. Once all of the reviews are in for a project (including that of the lead reviewer), the lead reviewer will read each of the reviews carefully, and then (via comments in EasyChair) attempt to clarify any disagreements between reviewers, and also to merge related points. They will take an active role creating a consensus among the reviewers. Once that is done, the lead reviewer will create a single "meta-review" that reflects the consensus view of the project, and submit that within EasyChair. The meta-review must be submitted by 6pm on the last day of term. I will then look over the meta-reviews, and once I'm happy with them I will send them to the project authors. Also, I will then look over your performance as a reviewer, and assign you a grade.
Since being a lead reviewer requires extra effort, they will have to review and discuss only one project. In Winter 2025, "normal" reviewers (i.e., everyone else) will be responsible to reviewing and taking part in the discussions of two projects.
You should structure your project reviews this way:
You will be able to see the other reviews on a project once you have submitted yours (but not before). Do not submit a blank review so you can peek at what others have said (I will be able to see if you do so, and I will not be happy). Note that the review online system does not allow you to delete a review or comment. However, you can revise a review; the old one will still be visible but will be "greyed out" so it's obvious that it's been superseded by a newer version.