Version 1
: Received: 5 May 2019 / Approved: 8 May 2019 / Online: 8 May 2019 (11:46:36 CEST)
Version 2
: Received: 15 August 2019 / Approved: 16 August 2019 / Online: 16 August 2019 (05:27:55 CEST)
How to cite:
Besançon, L.; Rönnberg, N.; Löwgren, J.; Tennant, J.; Cooper, M. Open Up: A Survey on Open and Non-anonymized Peer Reviewing. Preprints2019, 2019050098. https://doi.org/10.20944/preprints201905.0098.v2
Besançon, L.; Rönnberg, N.; Löwgren, J.; Tennant, J.; Cooper, M. Open Up: A Survey on Open and Non-anonymized Peer Reviewing. Preprints 2019, 2019050098. https://doi.org/10.20944/preprints201905.0098.v2
Besançon, L.; Rönnberg, N.; Löwgren, J.; Tennant, J.; Cooper, M. Open Up: A Survey on Open and Non-anonymized Peer Reviewing. Preprints2019, 2019050098. https://doi.org/10.20944/preprints201905.0098.v2
APA Style
Besançon, L., Rönnberg, N., Löwgren, J., Tennant, J., & Cooper, M. (2019). Open Up: A Survey on Open and Non-anonymized Peer Reviewing. Preprints. https://doi.org/10.20944/preprints201905.0098.v2
Chicago/Turabian Style
Besançon, L., Jonathan Tennant and Matthew Cooper. 2019 "Open Up: A Survey on Open and Non-anonymized Peer Reviewing" Preprints. https://doi.org/10.20944/preprints201905.0098.v2
Abstract
We present a discussion and analysis regarding the benefits and limitations of open and non-anonymized peer review based on literature results and responses to a survey on the reviewing process of alt.chi, a more or less open-review track within the CHI conference, the predominant conference in the field of human-computer interaction (HCI). This track currently is the only implementation of an open-peer-review process in the field of HCI while, with the recent increase in interest in open science practices, open review is now being considered and used in other fields. We collected 30 responses from alt.chi authors and reviewers and found that, while the benefits are quite clear and the system is generally well liked by alt.chi participants, they are reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.
Keywords
open review; open science; zero-blind review; peer review; methodology
Subject
Computer Science and Mathematics, Analysis
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Received:
16 August 2019
Commenter:
Lonni Besançon
Commenter's Conflict of Interests:
Author
Comment:
We have, based on a revision cycle, better justified our results presentations and the low number of respondents. Bold claims have been mitigated and related work discussed in greater length.
Received:
24 August 2019
Commenter:
Mario Malicki
Commenter's Conflict of Interests:
As an editor of RIPR, I would love to see this also published in our journal.
Comment:
Thanks for an interesting study, I left my comments also on Twitter @RIPRJournal
In section 3, I would welcome a description on how many submissions are there on average per year, and how many reviewers per submission, ideally accounting for how many reviewer reports were written by those submitting that year.
Please list the dates when u sent out the survey and that of the reminders. How many addresses did you send it to, what would be the response rate? Are authors in anyway involved with organisation of alt.chi, or did they ever submit or review for it (COI)?. Was there an ethics approval for the study?
In 4.1. I would advise adding initials of authors to the tasks they performed. Appendix A and Section A of the questionnaire are missing. Also, in the appendix, please list the number of answers per each question. What were the respondents’ answers to questions 17 and 18?
In section 4.1. if comments are divided into (possibly) positive and negative, how many fall into which category, and could you show this also per respondent.
For discussion - another possible exploration would be comparing reviewers of the open alt.chi with those of other tracks, in light of what issues are most commonly addressed, and in light of answers you present (e.g. stimulate the discussions between reviewers).
Commenter's Conflict of Interests:
First author of the paper
Comment:
Thank you very much for the feedback. We are now reviewing your comments and addressing them to submit to RIPR as the manuscript really seems to fit there.
We have now added some clarifications on the response rate (although it cannot be exactly computed because the link was also shared on twitter), we have now added how many emails were sent originally.
Concerning the dates, they have also been added (although reminders were only sent on twitter).
Thanks for noticing the problem with Appendix A. It was a problem in the original .tex file that we have now fixed.
Dividing comments into positive and negative ones is somehow difficult as some respondents mixed them. However, all our raw data is available online for more analysis.
We initially also considered comparing the feedback of people who review for alt.chi and people who review for other tracks, but our dataset contains only reviewers of all tracks (alt. + others) so we cannot include such a comparison in our discussion.
Comment 3
Received:
9 October 2019
Commenter:
Chat Wacharamanotham
The commenter has declared there is no conflict of interests.
Comment:
May be it would be more helpful to also describe the nature of the alt.chi track and its implication on the content and acceptance decisions.
In the pre-print, I only found the following:
'the alt.chi track within the CHI conference. Its initial purpose was to offer rejected papers a second chance through another round of peer-reviewing with new reviewers." [section 1 and 3]
The alt.chi venue is designed for controversial, risk-taking, and boundary pushing content, which may innovate methodologically, critique accepted practices, or take on controversial questions [CHI2020 website]. Therefore, the nature of the submitted content of alt.chi welcome discussions. To my knowledge, juries also take into account the likelihood of the diversity in opinion that a submission will generate as one of the criteria to accept an alt.chi submission.
This nature has three implications:
Rigor is not necessarily the main criteria of acceptance. In fact, there are a large number of alt.chi submission that are purely an essay or opinion.
Reviews that each alt.chi submission receives does not necessarily focus on scientific quality (e.g., internal validity, external validity, or transparency). From my personal experience of several years reviewing for alt.chi, I found that some good submissions (that could have been accepted by the normal paper track) are
not' accepted to alt.chi because it is not boundary-pushing enough.
The non-anonymous review process may also influence potential reviewers' decision of whether to add their review or add a comment on each submission.
Because of these implications, the following suggestion could be detrimental to the purpose of the venue:
"having a fixed number of assigned reviewers could solve many of the issues mentioned with the alt.chi system." [Conclusion section]
Commenter: Lonni Besançon
Commenter's Conflict of Interests: Author
Commenter: Mario Malicki
Commenter's Conflict of Interests: As an editor of RIPR, I would love to see this also published in our journal.
In section 3, I would welcome a description on how many submissions are there on average per year, and how many reviewers per submission, ideally accounting for how many reviewer reports were written by those submitting that year.
Please list the dates when u sent out the survey and that of the reminders. How many addresses did you send it to, what would be the response rate? Are authors in anyway involved with organisation of alt.chi, or did they ever submit or review for it (COI)?. Was there an ethics approval for the study?
In 4.1. I would advise adding initials of authors to the tasks they performed. Appendix A and Section A of the questionnaire are missing. Also, in the appendix, please list the number of answers per each question. What were the respondents’ answers to questions 17 and 18?
In section 4.1. if comments are divided into (possibly) positive and negative, how many fall into which category, and could you show this also per respondent.
For discussion - another possible exploration would be comparing reviewers of the open alt.chi with those of other tracks, in light of what issues are most commonly addressed, and in light of answers you present (e.g. stimulate the discussions between reviewers).
Commenter:
Commenter's Conflict of Interests: First author of the paper
We have now added some clarifications on the response rate (although it cannot be exactly computed because the link was also shared on twitter), we have now added how many emails were sent originally.
Concerning the dates, they have also been added (although reminders were only sent on twitter).
Thanks for noticing the problem with Appendix A. It was a problem in the original .tex file that we have now fixed.
Dividing comments into positive and negative ones is somehow difficult as some respondents mixed them. However, all our raw data is available online for more analysis.
We initially also considered comparing the feedback of people who review for alt.chi and people who review for other tracks, but our dataset contains only reviewers of all tracks (alt. + others) so we cannot include such a comparison in our discussion.
Commenter: Chat Wacharamanotham
The commenter has declared there is no conflict of interests.
In the pre-print, I only found the following:
'the alt.chi track within the CHI conference. Its initial purpose was to offer rejected papers a second chance through another round of peer-reviewing with new reviewers." [section 1 and 3]
The alt.chi venue is designed for controversial, risk-taking, and boundary pushing content, which may innovate methodologically, critique accepted practices, or take on controversial questions [CHI2020 website]. Therefore, the nature of the submitted content of alt.chi welcome discussions. To my knowledge, juries also take into account the likelihood of the diversity in opinion that a submission will generate as one of the criteria to accept an alt.chi submission.
This nature has three implications:
- Rigor is not necessarily the main criteria of acceptance. In fact, there are a large number of alt.chi submission that are purely an essay or opinion.
- Reviews that each alt.chi submission receives does not necessarily focus on scientific quality (e.g., internal validity, external validity, or transparency). From my personal experience of several years reviewing for alt.chi, I found that some good submissions (that could have been accepted by the normal paper track) are
not' accepted to alt.chi because it is not boundary-pushing enough.Because of these implications, the following suggestion could be detrimental to the purpose of the venue:
"having a fixed number of assigned reviewers could solve many of the issues mentioned with the alt.chi system." [Conclusion section]
Commenter:
The commenter has declared there is no conflict of interests.