That Time We All Talked About Peer Review Together

Yesterday morning, I was pretty sure my greatest accomplishment for the day was going to be braving the blizzard (or PolarBearSharkNadoVortex or whatever we’re calling it now) to see the Pixies in Newark with some friends. I went to work, where every morning we spend about a half hour talking about all the papers that have come out that day on astro-ph (if you’re not an astronomer, this is what I’m talking about). Astro-ph Coffee is one of my favorite things about Princeton– it’s short, informal, has a lot of participation by the faculty, and since it happens every day, it’s a great way of keeping up on what’s happening in the field (outside of one’s own subfield in particular). It also occasionally dips into entertainment, as a number of people in our department are possessed of the deadly combo of extensive knowledge and rapier wits. The discussion is largely about science, but once in a while, someone throws a mic-dropping gem of devastating commentary into the mix.

When that happened in yesterday’s discussion, I was reminded of the (possibly apocryphal) story about Hemingway being challenged to write a story in only six words– so I left the following sitting out on the internet:

and went about science-ing for a bit. When I looked again, well… I was going to Storify it, but there are literally hundreds of tweets.

Here’s a live feed of the hashtag, filtered for spam:

So, wow… do we all need a cup of tea and a sit down or what? It’s not like I was unaware of the ongoing debate/discussion about improving academic publishing and peer review, but I have to admit I’m surprised by just how much steam everyone needed to blow off. The common themes are not surprising, and many tweets fall into a few big bins (where some are more disturbing that others), e.g. “This paper didn’t cite me enough”, “You’re grammer its bad”, “Your institution isn’t fancy enough”, “My unpublished data refutes your result”, “I am only pretending to be qualified to review this paper”, etc. Amidst the hilarity, I think the following two tweets nicely encapsulate my thoughts at the end of the day:

I particularly like Alice Bell‘s call to action, because I think a fair amount about how we could make reviews better for everyone, and also what the role of anonymity is in the review process. Certainly this week, we saw a pretty clear example of how anonymity plays differently depending on privilege, where pseudonymous science blogger Dr. Isis was “outed” by Nature editor Henry Gee as a means of dismissing and intimidating her (here is a post by Dr. Isis on what happened, and here is further commentary by Michael Eisen). While in principle I like the idea of reviews being completely open, in many cases it is “safer” for early career researchers to critique the work of their more senior colleagues from behind the veil of anonymity, particularly if the review is negative in any way. Double blind reviews are another great idea and have wide support within the scientific community: Nature GeoSci went double-blind last summer, with somewhat mixed-but-evolving results. Practically speaking, though, I think that blindness is likely to be imperfect– if you are qualified to review one of my papers, you are likely to recognize my research (I also have a fairly distinctive writing style– seriously, I so overuse em dashes my emails are like really boring Emily Dickinson poems).

One idea for peer review I’ve been tossing around comes not from science, but from creative writing: face-to-face group reviews, usually known just as “critiques”. I first encountered critiques in college, where many of my electives were in fiction writing. Here’s how it works: your writing gets distributed to the class, and one of your peers serves as a primary reviewer who kicks off the discussion (sound a bit like proposal reviews, scientists?). Horrifyingly, you have to sit there with your mouth shut (completely shut, no talking, no rebuttal, nothing) while your peers then dissect and discuss your work in front of you– much like they might do if you were not in the room. The reasoning behind this is, of course, that you are essentially never going to be in the room with your reader to correct them on how they should be interpreting your work. The professor was present (though usually hands-off), so people made an effort to be constructive, and the conversation could be steered if it went too far off course. Only after the class had thoroughly discussed the work was the author allowed to speak, to either comment on what they had been trying to do, and/or to ask questions on how to improve the piece. It was a pretty mortifying experience as a young writer, but I think it also improved my work– and what’s more, it taught me how to critique other people’s writing while simultaneously helping me with my own.

So how might this work in academic publishing? Imagine first that having multiple reviewers was a standard practice for all journals (in astronomy, most journals have only a single referee)– let’s say three. Rather than soliciting written reviews from all reviewers up front,  imagine instead that you committed to spending an hour or so discussing the paper via Skype/Google Hangout/phone. When I’ve floated this idea to people in the past, the idea of having an actual discussion is sometimes met with groans– but I suspect that actually devoting that time to discussion rather than silo’d reading of a paper on your own being the only form of review might actually result in a more efficient process. For one thing, multiple reviewers would be able to bring the totality of their backgrounds to bear, sidestepping holes in the knowledge of one reviewer or another (holes that remain when multiple reviews proceed in parallel without discussion). Discussions could be on the record, and editors could even moderate if needed. If one were to really keep to the analogy with creative writing critiques, the lead author could also listen in on the discussion (and/or ask questions afterwards). Via voice alone (no video), I think this could actually be almost blind, though of course certain things like gender might be obvious. Another possibility would be analogous to proposal reviews, where one of the reviewers serves as a primary, collect the other reviewers’ comments and generating a written report of revisions to be distributed to the authors.

Like most of the suggested changes to the peer review process, implementing something like the above would take a cultural shift in our respective fields, and certainly a community demand for change. But given how many people had something to say about peer review today, it does seem to me that the momentum is at least a good part of the way there. The question is, what do we want to do about it?

That’ll take more than six words.

Worst. Superpower. Ever.

Worst. Superpower. Ever.

About these ads
This entry was posted in academia, academic publishing, open science, science, social media. Bookmark the permalink.

5 Responses to That Time We All Talked About Peer Review Together

  1. Matthew Nichols says:

    Some good ideas there. I like the concept of a live discussion, but there are some potential issues that may need to be tackled as well (although there will always be, and in the end it may be worth the drawbacks).
    The commitment of refereeing a paper will increase substantially, not only would people be doing it more often, but you now need to coordinate with two others to find a time.
    It also means that referees are going to be geographically clumped (for any individual paper).
    If you only have one or two groups in a region (say the Asia Pacific Region), people living there will likely rarely review, or discussions will have to take place at awkward hours which isn’t great for family balance.
    I’m also not sure voice alone would be that blind. My voice, for example, is somewhat broad even for Australians and is possibly recognizable to others that I would only meet rarely. Although I think voice would still be preferable (to avoid at least some what appearance being used to weight peoples opinions) it isn’t necessarily that blind, and so if you want blind referees, I would restrict the ability of the lead author to listen in. Even when written, referee reports can give hints as to who wrote them (how did they spell colour, did they make a mistake that is common to those whose first language is German, etc). A typed discussion may then be the best option, partially due to the lack of accents which hide some of it, and the automatic transcript that is generated (although still here, you have hints to the origin).

    Indeed if it is typed it doesn’t even have to be that live… maybe have the referees each write a review, then have a few days where people commit to looking at the other reports for say 10 minutes each day (of their own individual choosing, so it would give 4 chances to fix something). It would help solve the background knowledge etc. If the reports converge, one report is given otherwise the editor browses it and makes some decisions about what to do.

    • Thanks for the comments! I think if referees committed to reading and preparing initial comments on a paper before the live discussion, it would actually expedite the process overall. That is the standard practice for proposal review panels in astronomy, for example, where all reviewers prepare first draft written comments, but then only discuss the proposal for somewhere around 30 minutes (as there are typically lots and lots of proposals to review in a single day or few days). I would posit that reviewers are already geographically clumped (especially if you count only having a single reviewer as a clump!). Given that in many fields, having 2-3 reviewers is standard practice, I’m not sure that it would tax the community that much more– those other fields make it work, after all, and usually when I hear complaints about multiple referees it’s that there is no crosstalk between the reviews so opinions may vary wildly.

      Voice is definitely not completely blind (I think my voice is very recognizable, for example), but then maybe that’s not entirely a bad thing if the goal is to be open. But again, openness is riskier for some than others.

    • eLife is actually supposed to do this, right?…

  2. This is kind of what us biomed folks are attempting to do with PubMed Commons. See http://www.ncbi.nlm.nih.gov/pubmedcommons/.

  3. So how might this work in academic publishing? Imagine first that having multiple reviewers was a standard practice for all journals (in astronomy, most journals have only a single referee)– let’s say three. Rather than soliciting written reviews from all reviewers up front, imagine instead that you committed to spending an hour or so discussing the paper via Skype/Google Hangout/phone. When I’ve floated this idea to people in the past, the idea of having an actual discussion is sometimes met with groans– but I suspect that actually devoting that time to discussion rather than silo’d reading of a paper on your own being the only form of review might actually result in a more efficient process.

    I agree! 1 hour to read the paper and take notes, another hour to discuss it. 2 hours and you’re done! This would be a time-saving advantage to me, as a referee (or maybe I’m just a really slow referee…?), and as an author, I would also appreciate it as it would give me a chance to respond to criticisms and clarify confusing points.

    I would also advocate for more refereeing to become public instead of anonymous.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s