Yesterday morning, I was pretty sure my greatest accomplishment for the day was going to be braving the blizzard (or PolarBearSharkNadoVortex or whatever we’re calling it now) to see the Pixies in Newark with some friends. I went to work, where every morning we spend about a half hour talking about all the papers that have come out that day on astro-ph (if you’re not an astronomer, this is what I’m talking about). Astro-ph Coffee is one of my favorite things about Princeton– it’s short, informal, has a lot of participation by the faculty, and since it happens every day, it’s a great way of keeping up on what’s happening in the field (outside of one’s own subfield in particular). It also occasionally dips into entertainment, as a number of people in our department are possessed of the deadly combo of extensive knowledge and rapier wits. The discussion is largely about science, but once in a while, someone throws a mic-dropping gem of devastating commentary into the mix.
When that happened in yesterday’s discussion, I was reminded of the (possibly apocryphal) story about Hemingway being challenged to write a story in only six words– so I left the following sitting out on the internet:
and went about science-ing for a bit. When I looked again, well… I was going to Storify it, but there are literally hundreds of tweets.
Here’s a live feed of the hashtag, filtered for spam:Tweets by @shaka_lulu
So, wow… do we all need a cup of tea and a sit down or what? It’s not like I was unaware of the ongoing debate/discussion about improving academic publishing and peer review, but I have to admit I’m surprised by just how much steam everyone needed to blow off. The common themes are not surprising, and many tweets fall into a few big bins (where some are more disturbing that others), e.g. “This paper didn’t cite me enough”, “You’re grammer its bad”, “Your institution isn’t fancy enough”, “My unpublished data refutes your result”, “I am only pretending to be qualified to review this paper”, etc. Amidst the hilarity, I think the following two tweets nicely encapsulate my thoughts at the end of the day:
I particularly like Alice Bell‘s call to action, because I think a fair amount about how we could make reviews better for everyone, and also what the role of anonymity is in the review process. Certainly this week, we saw a pretty clear example of how anonymity plays differently depending on privilege, where pseudonymous science blogger Dr. Isis was “outed” by Nature editor Henry Gee as a means of dismissing and intimidating her (here is a post by Dr. Isis on what happened, and here is further commentary by Michael Eisen). While in principle I like the idea of reviews being completely open, in many cases it is “safer” for early career researchers to critique the work of their more senior colleagues from behind the veil of anonymity, particularly if the review is negative in any way. Double blind reviews are another great idea and have wide support within the scientific community: Nature GeoSci went double-blind last summer, with somewhat mixed-but-evolving results. Practically speaking, though, I think that blindness is likely to be imperfect– if you are qualified to review one of my papers, you are likely to recognize my research (I also have a fairly distinctive writing style– seriously, I so overuse em dashes my emails are like really boring Emily Dickinson poems).
One idea for peer review I’ve been tossing around comes not from science, but from creative writing: face-to-face group reviews, usually known just as “critiques”. I first encountered critiques in college, where many of my electives were in fiction writing. Here’s how it works: your writing gets distributed to the class, and one of your peers serves as a primary reviewer who kicks off the discussion (sound a bit like proposal reviews, scientists?). Horrifyingly, you have to sit there with your mouth shut (completely shut, no talking, no rebuttal, nothing) while your peers then dissect and discuss your work in front of you– much like they might do if you were not in the room. The reasoning behind this is, of course, that you are essentially never going to be in the room with your reader to correct them on how they should be interpreting your work. The professor was present (though usually hands-off), so people made an effort to be constructive, and the conversation could be steered if it went too far off course. Only after the class had thoroughly discussed the work was the author allowed to speak, to either comment on what they had been trying to do, and/or to ask questions on how to improve the piece. It was a pretty mortifying experience as a young writer, but I think it also improved my work– and what’s more, it taught me how to critique other people’s writing while simultaneously helping me with my own.
So how might this work in academic publishing? Imagine first that having multiple reviewers was a standard practice for all journals (in astronomy, most journals have only a single referee)– let’s say three. Rather than soliciting written reviews from all reviewers up front, imagine instead that you committed to spending an hour or so discussing the paper via Skype/Google Hangout/phone. When I’ve floated this idea to people in the past, the idea of having an actual discussion is sometimes met with groans– but I suspect that actually devoting that time to discussion rather than silo’d reading of a paper on your own being the only form of review might actually result in a more efficient process. For one thing, multiple reviewers would be able to bring the totality of their backgrounds to bear, sidestepping holes in the knowledge of one reviewer or another (holes that remain when multiple reviews proceed in parallel without discussion). Discussions could be on the record, and editors could even moderate if needed. If one were to really keep to the analogy with creative writing critiques, the lead author could also listen in on the discussion (and/or ask questions afterwards). Via voice alone (no video), I think this could actually be almost blind, though of course certain things like gender might be obvious. Another possibility would be analogous to proposal reviews, where one of the reviewers serves as a primary, collect the other reviewers’ comments and generating a written report of revisions to be distributed to the authors.
Like most of the suggested changes to the peer review process, implementing something like the above would take a cultural shift in our respective fields, and certainly a community demand for change. But given how many people had something to say about peer review today, it does seem to me that the momentum is at least a good part of the way there. The question is, what do we want to do about it?
That’ll take more than six words.