Varun Sivaram

Energy, Security, and Climate

CFR experts examine the science and foreign policy surrounding climate change, energy, and nuclear security.

Print Print Cite Cite
Style: MLA APA Chicago Close


Yellow Flags On A New Methane Study

by Michael Levi
February 13, 2012


A forthcoming paper in the influential Journal of Geophysical Research has measured concentrations of methane and other alkanes in air near oil and gas operations in Weld County, Colorado, and has used that to infer rates of methane leakage from natural gas production in the region. Their results, which were reported last week by Nature and the Associated Press, point to much higher methane emissions than consensus inventories have previously found. Indeed some have read the results as confirming the highly controversial estimates of methane leakage published last year by Cornell professor Robert Howarth and his colleagues.

The new study is far more sophisticated and careful than what’s been published so far. It deserves serious attention. But after several days of working through the data and analysis, I’m skeptical of some of the results and interpretations. That doesn’t mean that I think that the paper is wrong – it’s premature to say that – but, given the attention it’s been getting, I think it’s worth putting some of the potential problems out in public.

One note before I dive in: Some of my observations may seem a bit oblique. Recent experience has started to convince me that, when there’s a problem with peer-reviewed scientific work, it’s important to correct the formal scientific record, rather than just responding on the web. Alas, many scientific journals have bars on publishing material that’s already been made public elsewhere. In this case, I may end up submitting a technical “Comment” to JGR, so I’m going to have to be careful not to preempt myself here. I apologize to readers for whom that makes this post difficult to evaluate.

Now on to the four issues that I want to flag. The first two have to do with the paper itself; the second two are problems with how it’s being interpreted.

Assumptions about the composition of leaked gas.

The authors need to make some assumptions about the ratio of methane to propane in vented gas in order to do their analysis. They do that in three different ways in order to bracket uncertainty. There’s no reason, however, to assume that their three cases bracket the range of reasonable possibilities.

Alas, it turns out that their results are highly sensitive to the choice of ratio. If, for example, the methane-to-propane ratio in vented gas is a lot higher that in any of their three candidates, the implied rate of methane leakage is much lower than what they estimate. If, on the other hand, the methane-to-propane ratio in vented gas is lower than in any of their three candidates, the implied rate of methane leakage is even higher than their alarming estimates. This, to me, is the biggest problem in the paper, and introduces a lot of uncertainty around the results.

Are we really talking about gas wells?

The paper presents its final results as if it’s found out that there’s a lot of leakage from natural gas production. But a large fraction of the wells it uses to come up with chemical profiles for vented gas aren’t gas wells. In particular, the sorts of methane-to-propane ratios that it uses aren’t actually representative of gas wells – they’re representative of condensate (basically oil) wells with significant amounts of associated gas. In won’t get into the weeds here, but when I rework the numbers, I come to one of two conclusions: there’s a lot of methane leakage in the area they’ve studied, and it’s coming from condensate wells, or there isn’t much methane leakage there, but it’s coming from gas wells.

Why isn’t this just nitpicking? Because people are already taking the paper and trying to extrapolate it to genuine gas wells in places like Pennsylvania. It’s far from clear, though, that one can do that, if we aren’t actually talking about gas wells in the first place.

A peculiar spin on uncertainty.

The paper presents several different estimates of methane leakage (based on different methodologies) all with clearly reported uncertainties. Reporting has emphasized the full range of possibilities. But the range over which the different estimates overlap is actually pretty small, and it is close to the lowest, and least alarming, possible methane leak rates. It’s not clear to me why the upshot of this paper (setting aside the other issues I’ve flagged) isn’t that leakage rates are probably in this lowish area.

To understand why I’m puzzled, let me sketch an analogous example. Through decades of work, we know that the mass of the neutron is about 1.67E-27 kg. Now let’s say that I come up with a new experiment that shows that the mass of the neutron is between 1.5E-27 kg and 3.34E-27 kg. No one should write an article declaring “Mass of Neutron May Be Twice Previously Agreed Level”. They should look at my work and say: “Given that other methods peg the mass of the neutron at 1.67E-27 kg, and this new experiment is consistent with that, we should probably stick to what we know”. They should chalk the upper end of my experiment’s range up to the fact that my experimental method happens to be imprecise.

The same is true for this paper. Just because one approach in the paper can’t chop uncertainty down doesn’t mean that methane emissions might be stratospheric – it means that you should juxtapose that approach with other ones in order to narrow down the range of possibilities.

Apples, oranges, and the Howarth study.

The authors of the JGR paper report a “best guess” estimate that 4% of Weld County gas production is leaked. Howath tells Nature, “I’m not looking for vindication here, but [the JGR] numbers are coming in very close to ours, maybe a little higher”. Indeed the similarily between the JGR and Howarth numbers is one of the main reasons that the JGR paper has attracted attention.

But this comparison doesn’t work. In Howarth’s percentage calculation, the denominator is expected gas production from wells drilled during the period under study; in the JGR paper, that denominator is gas production during the period under study itself. These are different numbers, which means we’re talking about different ratios.

Bottom line? There’s a lot of fantastic observational data in the JGR paper. I suspect that there’s more that can be done with it to shed light on methane leakage. For now, though, I’m not ready to rely on its results.

Post a Comment 2 Comments

  • Posted by pjc

    “But this comparison doesn’t work. In Howarth’s percentage calculation, the denominator is expected gas production from wells drilled during the period under study; in the JGR paper, that denominator is gas production during the period under study itself. These are different numbers, which means we’re talking about different ratios.”

    This paragraph deserves further detail

    Isn’t the implication here that the JGR paper represents a more comprehensive estimate, and thus when compared with Howarth, the upper end of the estimates is still quite a bit lower than Howarth.

    I.e. Howarth is estimating the release from “just one stage” of the natural gas extraction process, getting a high estimate, adding it to other leakage sources, and getting a large total release.

    JGR is estimating the release from “the full extraction process”. So if this number matches Howarth, than it implies that the JGR estimate of the release from dilling is much smaller than Howarth’s number.

    Am I the only one who finds a resemblance here to the Keystone emissions debate? There is a similar “apples-to-oranges” mistake in comparing a sub-process to the whole process.

    The difference is the Keystone fanatics are mostly celebrities, whereas Howarth ought to know better.

  • Posted by David B. Benson

    I seriously doubt that natural gas constiuent percentages are constant across all gas fields, associated or not. Even after refining the natgas from your supplier has varying ratios:

Post a Comment

CFR seeks to foster civil and informed discussion of foreign policy issues. Opinions expressed on CFR blogs are solely those of the author or commenter, not of CFR, which takes no institutional positions. All comments must abide by CFR's guidelines and will be moderated prior to posting.

* Required