Search Episodes
Listen, Share, & Support
Listen to the latest episode
Subscribe via iTunes
Subscribe via RSS
Become a fan
Follow on Twitter

CHOOSE MEMBERSHIP LEVEL

E-mail Updates

Enter your e-mail address to receive updates about the Rationally Speaking podcast and NYC Skeptics

Related Readings
  • Answers for Aristotle: How Science and Philosophy Can Lead Us to A More Meaningful Life
    Answers for Aristotle: How Science and Philosophy Can Lead Us to A More Meaningful Life
    by Massimo Pigliucci
  • Nonsense on Stilts: How to Tell Science from Bunk
    Nonsense on Stilts: How to Tell Science from Bunk
    by Massimo Pigliucci
  • Denying Evolution: Creationism, Scientism, and the Nature of Science
    Denying Evolution: Creationism, Scientism, and the Nature of Science
    by Massimo Pigliucci
Tuesday
Mar202012

RS57 - Peer Review

Release date: March 25, 2012


If you value scientific evidence you're probably familiar with the idea that having "peer-reviewed" studies is crucial to the legitimacy of any new claim. But what does "peer-reviewed" entail, anyway? In this episode, Massimo and Julia open up the black box of peer review, explaining how the process originated, how it works, and what's wrong with it. They also try brainstorming ways it could be fixed, and ask: how is the Internet changing the way we do research?

Julia's pick: The game "Zendo"
Massimo's pick: "Download The Universe"

References:

www.insidehighered.com/news/2012/01/06/humanities-scholars-consider-role-peer-review

References (1)

References allow you to track sources for this article, as well as articles that were written in response to this article.

Reader Comments (11)

Thank you for this interesting episode.

I think you misrepresented the role of arXiv a little bit. It's true that some people post versions of their papers that have not yet made it through peer review (or never will). However, at least in the more traditional astronomical fields, the standard is to wait for a paper to be officially accepted, and then put the manuscript on arXiv as soon as possible. Usually the authors use the "comments" field on each submission to indicate whether a paper has actually passed peer review (or, at least, whether it has been submitted to a journal). Since, even after official acceptance, the post-review editing process can take up several weeks to months, there is still a great benefit to putting this pre-print on the net for the community to read.

arXiv also has another advantage: usually the arXiv version of one's peer reviewed paper can be updated to take into account changes made during the final editing stages. This has the effect that, eventually, there is an open-access version of the paper available which is more or less identical in content (except for some minor differences in typesetting and layout) to the version published in the restricted-access journal. So, if you are interested in a paper from a few years ago which is still behind a pay wall, there is a good chance that arXiv provides an open-access copy.

Of course, some journals don't allow the author(s) to release any open-access version of work, and others try to control the information flow using embargos. Even though some of these journals are very highly regarded, I don't think researchers are very fond of these restrictions.

Best regards,
Michael

March 26, 2012 | Unregistered CommenterMichael Gruberbauer

Perhaps Massimo's pick was intended to link to "http://www.downloadtheuniverse.com/" ? At the moment it is linked to "http://bit.ly/GSMjMC" which resolves to "http://www.koryheath.com/games/zendo/"

HTH

March 27, 2012 | Unregistered CommenterAlan Brogan

Thank you HTH, the link is now fixed.

March 27, 2012 | Registered CommenterAdmin

Julia asked why are not all reviews blind and there is an answer for that. If people know who reviewed their papers they usually do not use mean spirit comments. I saw many nasty and personal commets that were not very professional from annonymous reviewers.

March 28, 2012 | Unregistered CommenterGil

Interesting podcast. You are probably not aware of our venture for biomedical scientists. We publish all content in an environment of post publication peer review on WebmedCentral and have been doing so for nearly 18 months now. It has been an interesting ride so far and we do have nearly a thousand articles, 1.5 times the reviews and more than 4000 registered users. If you have time, please check us out. All constructive feedback welcome.

Kind regards,

Kamal Mahawar
CEO, Webmed Limited, UK

April 1, 2012 | Unregistered CommenterKamal Mahawar

My month-old gobbled-up post on the -isms episode finally published today! Something went wrong, obviously.

April 3, 2012 | Unregistered CommenterMark Erickson

Similar to arXiv for social sciences is SSRN.

One type of peer review that you didn't discuss is the Open Peer Commentary practiced originally by Current Anthropology and also by Behavioral & Brain Sciences. Rather than just publishing an article, these journals invite a fairly large number of reviewers to write reviews/critiques/commentaries on the articles and then publish those and a response by the author(s) along with the original article. This obviously takes an enormous amount of work, but it gives the reader a really good picture of the status of the claims in the article, their context in a bigger picture, possible weaknesses, and how they are regarded. Behavioral and Brain Sciences is also highly interdisciplinary, which means commentary comes from philosophers, psychologists, brain scientists, computer scientists, anthropologists, and so on.

From the description of the podcast, I was also hoping to hear a bit more about the origins of peer review--you didn't really talk a whole lot about how the specific methods of scientific publication became established by the scientific societies in the first place. A bit of that history with respect to the Royal Society is recounted in Steven Shapin's fascinating book, _A Social History of Truth_. How things get appropriately registered as objective is a topic that has been discussed by Peter Galison, Simon Schaffer, Ted Porter, and Ian Hacking, among others.

April 9, 2012 | Unregistered CommenterJim Lippard

FYI: Ray Spier, "The history of the peer review process," _TRENDS in Biotechnology_ vol. 20, no. 8, August 2002, pp. 357-358: http://144.206.159.178/ft/1057/64598/1102063.pdf

June 15, 2012 | Unregistered CommenterJim Lippard

I'am afraid that the post misses the point. At least in scienctific disciplines, the decision to publish a paper or not is up to the editor, not to the reviewers. The editor can decide even not to send the paper in revision (this often occurs) and he cen theoretically even decide contrary to the revieweres (although this never occurs, of course). The role of the reviewers is to improve the article. In my experiende.,when my paper was rejected the journal was perfectly right, althought it's hard to accept.

September 11, 2012 | Unregistered CommenterGiuliano Fanelli

I don't know if this comment will be read, but I just recently started listening to the podcast and been going through the archive.

First, about the non-blinding of authors. In many cases it would any way be obvious who the principal investigator is, either by looking at the reference list, or, in many fields, by the fact that only very few principal investigators could have carried out the work (special equipment, etc.) Also, in a perfect world, ideas would indeed be judged on their own merit. By in the real world, a referee has limited time and resources to devote to the process. Knowing who the authors are can be very helpful: some researchers I know to be very careful and meticulus, so I can take it at face value when, for instance, they say they have performed test x to confirm point y. Others I konw to be sloppy, and I will pay more attention to some details because of this.

Second, about the publication of the refrees' name at publication. You seem to have only considered the case where either the paper was good, and therefore the interaction with the referee was cordial, or the paper was bad, and was rejected so the referee will stay anonymous. What about when there a battle between the referee and the authors during the peer review, but the article still ends up being published? You think the work is not good, another referee thinks it is good, the editor decides to publish anyway, do you still want your name to be revealed?

March 13, 2013 | Unregistered CommenterClaude

On the issue of the bogus journals in the Pharma sector, looks like it was Elsevier ...

http://www.the-scientist.com/?articles.view/articleNo/27383/title/Elsevier-published-6-fake-journals/

No less than six journals between 2000 and 2005 (!!)

August 15, 2013 | Unregistered CommenterClint Goss

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>