CFR Presents

Energy, Security, and Climate

CFR experts examine the science and foreign policy surrounding climate change, energy, and nuclear security.

Print Print Email Email Share Share Cite Cite
Style: MLA APA Chicago Close

loading...

The Peer Review Fetish

by Michael Levi
July 8, 2011

The following statement in a ClimateProgress post this past weekend jumped out at me:

“Peer review is the basis of modern scientific endeavour.”

This strikes me as an accurate reflection of the way that many – perhaps most – in the climate world think. It’s also wrong.

The sentiment is not restricted to one side of the policy fight. Yes, climate campaigners regularly point out that the work that backs them up is peer reviewed. Yet many of last year’s attacks on the IPCC (and often, by extension, the enterprise of addressing climate change) focused on its use of non-peer-reviewed sources too.

Why is the statement wrong? Modern science rests on a mix of transparency, replicability, and peer evaluation and challenge. Huge numbers of (sometimes peer-reviewed) papers get published. Some of them stand the test of time. Others don’t. What separates the wheat from the chaff is ultimately whether the work withstands broad scrutiny.

Peer review is only a small part of this dynamic. Journals (and book editors) make a point of sending papers out to two or three people for (usually blind) review. Positive reviews indicate that the reviewers think that the paper is of sufficient quality for the journal that’s considering it. They do not necessarily indicate agreement with the paper’s conclusions. Indeed they do not even necessarily mean that the reviewers think that the paper is all that good: that depends on the journal it’s being considered for.

A couple reviewers, of course, are a poor substitute for mass scrutiny. Sometimes reviewers are chosen poorly; other times they’re lazy. For a complex, interdisciplinary paper, it’s often impossible to find reviewers who actually understand the whole thing.

Once a paper is published, the real evaluation begins. I would place a lot more trust in a paper that withstood assessment by hundreds of other researchers after publication, even if it was not initially peer reviewed, than one that was published in a peer reviewed journal but was subsequently trashed by large numbers of competent researchers. Better yet, I’d look for both peer review and widespread acceptance.

This, however, does not appear to be how most people in the climate world think. Journalists are too often allowed to turn off their brains once they’re handed a piece of peer reviewed work (or told that another piece of work has not undergone peer review). Peer review converts the paper into gospel; all that remains is to preach its existence to the masses. Conversely, to attack a good study that is not peer reviewed, all that seems to be required is an attack on that procedural shortcoming; substance, too often, becomes secondary.

I can understand why peer review has become the gold standard in some quarters. The IPCC, in particular, needs a simple screen for quality, given the immense sprawl of its activities. And, to be fair, many deeply flawed studies only see the light of day because they avoid the barrier of peer review. That, however, merely indicates that peer review is often a useful minimum standard. It shouldn’t be read as anything more.

This is not an attack on the importance of science, or rigorous argument, in informing climate policy. Quite the opposite. Conflating peer review with scientific soundness impoverishes our appreciation of the scientific process. Peer review should be one criterion that people use in assessing the strength of any given piece of research – nothing more, nothing less.

Post a Comment 5 Comments

  • Posted by Purgatus

    Thoughtful, sobre, and obviously true statements such as this are why I come back to this blog over and over again.

    Too often it is impossible (or very nearly so) to take nuanced positions on ANYTHING climate related, even on such basic concepts as the process of knowledge aquisition itself.

  • Posted by Purgatus

    The other side of this coin is the complete rejection of peer review that is often indulged in by AGW skeptics. It’s a feedback loop that comes from those on both sides’ extreme positions reinforcing the extreme positions of their opponents.

    “Peer review is gospel!” (Screams one side seeing the other question the peer review process).

    “Peer review is worthless!” (Screams the other side, seeing what they view as bad science being treated as fact post-peer review by the opposition).

    The study of polarization is a fascinating subject in and of itelf.

  • Posted by Duncan Kinney

    Bit of a linkbait headline for the denier crowd no? I don’t think you’d get much play if your hed was what you’re trying to say. “Peer review often a useful minimum guideline”

    [ML: I think you misread me. The minimum guideline thing was a caveat. The post was genuinely intended to a push-back against the obsession with peer review. If deniers/skeptics/whatever agree, so be it.]

  • Posted by Sallan Foundation

    Sure, good science is that work which is replicable & holds up over time. Peer review is a first step, not the sum total of that process. The role of peer reviewers, a/k/a gatekeepers in climate science should be kept transparent to shoot down the mud-slinging from skeptics; you’re right, peer review ought not to be a sacred cow.

    That said, your column, while worth reading, lapses into a straw man attack on how unnamed climate scientists work. You’ve got a beef with someone besides Joe Romm? then be specific. And, when you have instances of the “peer reviewed’ label putting a sock in the mouth of scientific critique, give us the evidence.

    [ML: I may do that later. I didn't want to selectively name names, for fear of distracting from the broader point. It's not difficult to round up examples.]

  • Posted by Jeff Wishart

    I think that you are misunderstanding just what the term “peer review” means. Of course it includes the initial reviewers acting as gatekeepers to the journal to which the author submitted, but those “hundreds of other researchers after publication” that review the article are also part of the peer-review process, albeit in an ad hoc and informal manner. Peer review in fact never ends: once an article is published in the public domain, it lives or dies depending on how its information stands up to further scrutiny and observation. If the findings of a paper are overturned by an experiment a hundred years on, that is still peer review. Restricting ‘peer review’ as you did in your article is doing it a disservice.

    As such, I agree with Sallan Foundation: this article is essentially one long straw-man argument. Nobody serious (and I think that Joe Romm can be included here) would believe that the initial reviews by a couple of people is the sum total of peer review that is necessary and that if an article passes it that it’s golden.

    The peer review process is far from perfect, but it’s important to support it despite its imperfections because otherwise you have intellectual chaos. It’s true that sometimes people place too much faith in the initial journal review, but it’s a lot better than the other side that eschews the peer review process entirely.

Post a Comment

CFR seeks to foster civil and informed discussion of foreign policy issues. Opinions expressed on CFR blogs are solely those of the author or commenter, not of CFR, which takes no institutional positions. All comments must abide by CFR's guidelines and will be moderated prior to posting.

* Required