Preprint Article Version 2 This version is not peer-reviewed

Open Up: A Survey on Open and Non-anonymized Peer Reviewing

Version 1 : Received: 5 May 2019 / Approved: 8 May 2019 / Online: 8 May 2019 (11:46:36 CEST)
Version 2 : Received: 15 August 2019 / Approved: 16 August 2019 / Online: 16 August 2019 (05:27:55 CEST)

How to cite: Besançon, L.; Rönnberg, N.; Löwgren, J.; Tennant, J.; Cooper, M. Open Up: A Survey on Open and Non-anonymized Peer Reviewing. Preprints 2019, 2019050098 (doi: 10.20944/preprints201905.0098.v2). Besançon, L.; Rönnberg, N.; Löwgren, J.; Tennant, J.; Cooper, M. Open Up: A Survey on Open and Non-anonymized Peer Reviewing. Preprints 2019, 2019050098 (doi: 10.20944/preprints201905.0098.v2).

Abstract

We present a discussion and analysis regarding the benefits and limitations of open and non-anonymized peer review based on literature results and responses to a survey on the reviewing process of alt.chi, a more or less open-review track within the CHI conference, the predominant conference in the field of human-computer interaction (HCI). This track currently is the only implementation of an open-peer-review process in the field of HCI while, with the recent increase in interest in open science practices, open review is now being considered and used in other fields. We collected 30 responses from alt.chi authors and reviewers and found that, while the benefits are quite clear and the system is generally well liked by alt.chi participants, they are reluctant to see it used in other venues. This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.

Subject Areas

open review; open science; zero-blind review; peer review; methodology

Comments (2)

Comment 1
Received: 16 August 2019
Commenter: Lonni Besançon
Commenter's Conflict of Interests: Author
Comment: We have, based on a revision cycle, better justified our results presentations and the low number of respondents. Bold claims have been mitigated and related work discussed in greater length.
+ Respond to this comment
Comment 2
Received: 24 August 2019
Commenter: Mario Malicki
Commenter's Conflict of Interests: As an editor of RIPR, I would love to see this also published in our journal.
Comment: Thanks for an interesting study, I left my comments also on Twitter @RIPRJournal

In section 3, I would welcome a description on how many submissions are there on average per year, and how many reviewers per submission, ideally accounting for how many reviewer reports were written by those submitting that year.

Please list the dates when u sent out the survey and that of the reminders. How many addresses did you send it to, what would be the response rate? Are authors in anyway involved with organisation of alt.chi, or did they ever submit or review for it (COI)?. Was there an ethics approval for the study?

In 4.1. I would advise adding initials of authors to the tasks they performed. Appendix A and Section A of the questionnaire are missing. Also, in the appendix, please list the number of answers per each question. What were the respondents’ answers to questions 17 and 18?

In section 4.1. if comments are divided into (possibly) positive and negative, how many fall into which category, and could you show this also per respondent.

For discussion - another possible exploration would be comparing reviewers of the open alt.chi with those of other tracks, in light of what issues are most commonly addressed, and in light of answers you present (e.g. stimulate the discussions between reviewers).
+ Respond to this comment

We encourage comments and feedback from a broad range of readers. See criteria for comments and our diversity statement.

Leave a public comment
Send a private comment to the author(s)
Views 0
Downloads 0
Comments 2
Metrics 0


×
Alerts
Notify me about updates to this article or when a peer-reviewed version is published.