The “Science” Behind Generating Headlines
From the draft report on water quality in Pavillion, Wyo., to a Cornell graduate student's paper on public health, opponents of oil and gas development are displaying a troubling habit of leaping to conclusions before even basic scientific review can be completed. But if your job is to generate headlines, why let science get in the way?
Last December, EPA released a draft report on water quality in tiny Pavillion, Wyo., which was immediately seized upon by opponents of natural gas development in the United States (and even around the world) as smoking-gun proof that hydraulic fracturing pollutes drinking water. Never mind that the paper hadn’t been peer reviewed, or that within a few months the EPA to backtrack and admit that its testing procedures were inadequate, suspending peer review altogether until new sampling could be completed. Just two months after the release of the draft report, EPA Region 8 administrator Jim Martin told a House panel in no uncertain terms that the agency had not established a “causal link” between hydraulic fracturing and water contamination.
Fast-forward to today. Shale opponents have now seized upon yet another “report” (from Cornell, where else?) that supposedly links poor infant health (specifically low birth weight) to natural gas production. And, once again, the paper has not yet undergone peer review — the very process that helps sort out, at least in theory, legitimate scientific conclusions from simple suppositions or even outright activism. In fact, left unmentioned by the activists cheering the release of the paper is the fact that the author, Elaine Hill, is a graduate student in applied economics and management — hardly a field that one would expect to include complex epidemiological assessments.
Andy Revkin at the New York Times – certainly no shill for the oil and gas industry! — has done a deep dive into the problem of jumping the gun on this kind of research, including the fact that opponents are now using Ms. Hill as some sort of “champion” of their cause. What Revkin uncovered, among many things, is that the activist group New Yorkers Against Fracking hired a PR firm (BerlinRosen Public Affairs) to promote the piece, and the firm sent out a pitch to the media about the paper, stating only that it was written by a “researcher at Cornell” — nothing about peer review, and nothing about the fact that the author is still a graduate student. Revkin asked BerlinRosen about why they were promoting a paper before peer review, to which the firm replied that Ms. Hill’s results raise “critical questions” that “should be discussed.”
Why is this worth noting? Because Ms. Hill herself told Mr. Revkin that her results are “preliminary” (that aspect was ignored by activists, either deliberately or inconveniently) and that she “does not want to rush it” in terms of publication. She also said it’s a “valid” point to suggest that her conclusions not be cited until peer review is complete — completely undermining the PR firm (on behalf of New Yorkers Against Fracking) that tried to do exactly the opposite.
EID did its own review of the paper earlier this week, and what follows is a list of some of the most significant concerns and items of interest:
Item I: Skewed results indicate more complexity
Based on Ms. Hill’s results, low birth weight increased by 25 percent for babies born to mothers living within 2.5 km of a well. But at 2.0 km, according to her research, the increase was 26 percent — and at 1.5 km the increase was 21 percent (p. 18). In other words, the closest sample actually had a lower rate of increase than the sample set farther away, with the middle actually recording the highest increase. If there were a “causal relationship” between natural gas development and low birth weight, as she herself claims (see Item II below), why didn’t her results show higher intensity closer to the well? After all, a “causal” relationship means “If A then B,” not “If A then maybe B.”
Item II: Working backwards from a conclusion
The author says up front that her research investigates the “causal relationship between unconventional NGD [natural gas development] and infant health in Pennsylvania” (p. 2). This is not just a strongly worded statement affirming the validity of your hypothesis before peer review (a dangerous game in and of itself), it’s also completely unscientific. When you compile the “first” assessment of anything (as she claims, see p. 4), you are essentially by any legitimate scientific definition not establishing a “causal” link. But Ms. Hill essentially establishes that link by decree. This could be poor phrasing, but given the conclusions later in the paper that stem from that statement and indeed hinge upon it (more on that below), this seems like more than simply a semantic oversight. (Interestingly, Ms. Hill told Revkin that her language was intentionally strong, but that it wasn’t meant to mislead readers about the “caveats surrounding these findings.”)
Item III: Building toward a convenient narrative
The author couldn’t help but leap from an empirical research project into the policy and advocacy realm, stating at the end of her paper (p. 21-22): “These results suggest that policies that intend to prevent pollution exposure stemming from unconventional natural gas development should increase the regulated/allowable distance between drilling activity and nearby residences.” She then reaches even further, saying that shale development is occurring in 31 states nationwide, which, to her, means that “there is likely to be many exposed babies resulting in a nationwide increase in LBW.” (Interestingly, Ms. Hill has already backtracked from this statement, stating earlier this week that her results actually have “limited external validity” — a complete contradiction of the assertion in her paper.)
Ms. Hill also states that since her paper only looked at the impacts on infant health at birth, “the total increased health costs due to unconventional natural gas development are likely to be much greater.” Again, no credible scientific study would start to make these kinds of extrapolations or recommendations based upon a single assessment that the author herself admits is essentially a first of its kind – unless (a) the purpose was to make those kinds of recommendations and then work backwards from there, or (b) she truly believes that a single initial assessment — limited in scope — scientifically justifies conclusions that require evidence that she has not even collected, much less analyzed.
Item IV: Flawed air pollution claims
Ms. Hill cites air pollution problems in the Dallas-Fort Worth area (p. 9-10), mentioning a single assessment conducted by a consulting group in 2011. What Ms. Hill doesn’t mention is the research that’s been done by the Texas Commission on Environmental Quality (TCEQ), the regulatory body in charge of regulating air emissions in the state, which also has access to the most comprehensive emissions data in Texas. Here’s what TCEQ has said about the region Ms. Hill references:
“After several months of operation, state-of-the-art, 24-hour air monitors in the Barnett Shale area are showing no levels of concern for any chemicals. This reinforces our conclusion that there are no immediate health concerns from air quality in the area, and that when they are properly managed and maintained, oil and gas operations do not cause harmful excess air emissions.”
Ms. Hill also claims there are “no current studies [examining air emissions and toxicities] in the Marcellus shale in Pennsylvania” (p. 10) – a categorically false claim. The Pennsylvania Department of Environmental Protection (DEP) conducted ambient air sampling reports for the Northeast Marcellus and the Southwest Marcellus, both under Gov. Ed Rendell (D). After extensive study, the DEP “did not identify concentrations of any compound that would likely trigger air-related health issues associated with Marcellus Shale drilling activities” in either assessment. Perhaps Ms. Hill was referring to peer-reviewed academic studies, but to pretend these comprehensive regulatory assessments simply don’t exist seems like a huge omission — and perhaps a little too convenient.
Item V: Irrelevant data proves health risks?
The paper cites a paper compiled by Cornell veterinarians (which itself was fundamentally flawed) as essentially scientific proof of her conclusions, even though she admits the lack of applicability: “Although their study is not an epidemiologic analysis, nor is it a study that identifies specific chemical exposures related to NGD, it provides evidence that there are clear health risks in natural gas development” (p. 11). This is an enormous logical fallacy: “A doesn’t provide evidence of B, but A nonetheless clearly proves B.” And remember, a member of the U.N. Environmental Programme called the veterinarians’ paper “an advocacy piece” that “does not qualify as a scientific paper.”
—
The upshot here is that a paper that has not yet undergone peer review should, as Lisa McKenzie of the Colorado School of Public Health has said, be “approached with extreme caution.” But for opponents of hydraulic fracturing and natural gas development, the need to spread information that maligns the oil and gas industry trumps even basic standards of fact checking, much less a lengthy scientific review process.
This is now the second time in less than a year that anti-shale activists have run with a conclusion that has not been scientifically vetted, and unfortunately for their credibility, it’s also the second time that even a cursory review of available data shows how flawed their conclusions truly were. And that’s in addition to the AP analysis that ran earlier this week, which showed how ideological opponents have seized upon convenient talking points, even when the facts completely contradict them.
*UPDATE* Cornell Veterinarians Go Into “Beast Mode” on Shale
When it comes to the issue of responsibly developing oil and natural gas resources from shale, we’ve seen a lot of wacky things come out of Ithaca, New York over the past couple years. So it was no surprise when a pair of veterinarians associated with Cornell wrote an article attacking shale development...
UPDATE (4/6/2012, 1:15pm ET): Some intrepid research by the EID team has uncovered a meaningful critique of the Bamberger-Oswald paper, and the source is no slouch: Dr. Ian Rae, a professor at the University of Melbourne in Australia and a Co-chair of the Chemicals Technical Options Committee for the United Nations Environment Programme, says the paper is “an advocacy piece” that suffers from poor referencing, and the authors themselves “cannot be regarded as experts” in the field in which they are commenting. Rae’s full comments about the paper can be found here, but we’ve excerpted the most significant items below:
- “It certainly does not qualify as a scientific paper but is, rather, an advocacy piece that does not involve deep…analysis of the data gathered to support its case.”
- “The data in Table 2 are incomplete in that no dates or places are provided, and no references to other commentary on the events it reports, so it’s hard to assess the weight of the evidence. Surely there were reports to or by regulatory agencies. It could be that this is old evidence and that note has been taken of the hazards and appropriate regulations put in place to mitigate them. We just don’t know.”
- “Contributions to the journal are said to be refereed, but the refereeing process evidently was not very stringent. For example, better refereeing would have forced the authors to provide the details I identified above as missing from their compilation. As well, it might also have curtailed some of the less-well supported statements and asked for more recent references to the scientific basis for expressions of concern that material dated to the 1960s and 1970s.”
- “As far as I can see, neither [Bamberger nor Oswald] has a track record of investigation in environmental studies. This does not mean they are wrong to sound a note of concern, but it does mean that they cannot be regarded as experts in the field with broad experience and attainments.”
- “I have not had time to read the articles in recent issues of the journal, but the titles show that they are advocacy pieces dealing with issues that are matters of concern, and for that reason are also extensively covered by other journals.”
—Original post from January 11, 2012—
When it comes to the issue of responsibly developing oil and natural gas resources from shale, we’ve seen a lot of wacky things come out of Ithaca, New York over the past couple years.
The primary recipient of millions of dollars every year of anti-shale advocacy provided by the Park Foundation (also based in Ithaca), Cornell University has become to anti-energy activists what “Linebacker U” was once to Penn State — with the debunked-ad-nauseum Howarth paper on shale emissions serving as the movement’s main playbook. Ithaca also happens to be the place from which outlets like the New York Times pull “data” on mineral leasing, notwithstanding the fact that no actual Marcellus development even takes place there.
So it was no surprise when a pair of veterinarians associated with Cornell wrote an article attacking shale development for its supposed link to animal health impacts. (One of the authors, Robert Oswald is a professor at Cornell’s College of Veterinary Medicine; the other, Michelle Bamberger, received her doctorate from Cornell.)
Now, needless to say, we don’t have any bones to pick with veterinarians, and in fact the scientific research they provide on a daily basis is without question critical to us better understanding the natural world (plus, we love dogs). But the authors here did not produce a scientific assessment, a fact they freely admit in their article. Instead, Oswald and Bamberger chose to highlight a handful of personal testimonials that cannot be independently assessed or verified because they decided to keep all relevant details anonymous. Thus, we’re left with a 27-page unscientific article making bold assertions about oil and gas development, without a single shred of data or independent corroboration to back any of it up.
While the article contains many flaws, we’ve highlighted a few of the key problems below, all of which should raise serious doubts about the “scientific” nature of this particular article.
- Right off the bat, the paper leads with a philosophical quote from Sandra Steingraber, who has described hydraulic fracturing as “the tornado on the horizon” that will destroy people’s ability to do everything, from local gardening to even riding a bicycle (Orion Magazine, Sept./Oct. 2010). Ms. Steingraber has also called for an end to all fossil fuels to “avoid human calamity.” With respect to shale development, Ms. Steingraber has stated: “If we mitigate fracking to kill fewer people, we’re still killing people” (The Vindicator, Jan. 10, 2012).
- The authors assert that developing natural gas from shale is “moving forward without benefit of carefully controlled studies of its impact on public health” (p. 52). Aside from the fact that the authors readily admit in the paper that their own conclusions are not the result of controlled experiments, their claim is simply not true. For example, a study from earlier this year by the city of Fort Worth, TX, concluded there were “no significant health risks” from nearby shale development (July 2011).
- A separate scientific assessment of the Barnett Shale in north Texas concluded: “[E]ven as natural gas development expanded significantly in the area over the past several years, key indicators of health improved across every major category during those times” (Oct. 19, 2011). The Barnett Shale is one of the most productive shale fields in the United States, with more than 15,000 producing wells.
- Instead of seeking out the answer to a legitimate question – what, if any, are the health impacts of developing natural gas from shale? – the authors simply accuse the industry of taking a position “similar to the tobacco industry that for many years rejected the link between smoking and cancer” (p. 52). The report goes on to suggest that “epidemiologic studies [that] linked smoking to human health impacts…could be used to assess the health impacts of gas drilling operations on human beings” (p. 53). It seems the authors have already made up their minds.
- The authors clearly admit that the study is not sound science: “This study is not an epidemiologic analysis of the health effects of gas drilling, which could proceed to some extent without knowledge of the details of the complex mixtures of toxicants involved. It is also not a study of the health impacts of specific chemical exposures related to gas drilling” (p. 53).
- Later in the article the authors further concede: “By the standards of a controlled experiment, this is an imperfect study, as one variable could not be changed while holding all others constant” (p. 55). Instead, the article is merely a compilation of unsourced and unverifiable case studies.
- The report conceals names and locations, which means independent review of the claims and parties involved cannot be completed; statements from the researchers about their findings are simply asserted as fact. Ironically, much of the paper is committed to critiquing the industry for not disclosing enough information to independently verify data.
- Despite its lack of scientific bent, the authors nonetheless conclude definitively that their assessment “strongly implicates exposure to gas drilling operations in serious health effects on humans, companion animals, livestock, horses, and wildlife.” They go further and, without any scientific evidence, state that “a ban on shale gas drilling is essential for the protection of public health” (p. 72).
Calling for a ban on responsible oil and gas development without any scientific basis? Wait, we’ve heard this one before…
Again, those interested in the supposed health impacts of developing natural gas from shale should reference this assessment from October, in which two public health professionals studied conditions in the Barnett shale region of north Texas. Their conclusion? Even though the area has been one of the highest gas producing regions of the country, “key indicators of health improved across every major category.” That followed a study from last summer for the city of Fort Worth which “did not reveal any significant health threats” from shale development.
There They Go Again: Latest Cornell Paper Just More of the Same
It was touted by their PR consultants in a media advisory sent around this week as “a major new paper” in response to the mountain of criticism that has accumulated since the release of their initial study last April (that is, after the first one the year before was retracted).
It was touted by their PR consultants in a media advisory sent around this week as “a major new paper” in response to the mountain of criticism that has accumulated since the release of their initial study last April (that is, after the first one the year before was retracted). But read through the “new” document released by Cornell professors Robert Howarth and Anthony Ingraffea on Thursday, and you quickly come to the following realization: Not only don’t these guys offer a credible response to any of their critics, they don’t even acknowledge that they exist.
Last April, EID took the lead on pointing out some of the more obvious errors in the Cornell GHG report, eventually (and happily!) making way for the cavalcade of academics, government agencies and even environmental groups that followed – each one successively identifying a new and seemingly more obvious and/or egregious miscalculation that, taken on its own, would render the entire thesis unacceptable.
Of course, because none of those basic errors were either corrected or acknowledged in the latest iteration of the Howarth paper, there’s just not a whole lot of new material for us to rebut. Still, a couple of points are probably worth making for those genuinely interested in understanding what’s going on here.
Point #1: The “new” Howarth paper attempts to justify its previous conclusions based on what it says are “new” emissions data from EPA.
The problem?
- Howarth et al. are correct that EPA issued new emissions estimates for natural gas systems in late 2011 – that document is here for anyone who would like to give it a look. Unfortunately for the Cornell researchers, though, those new estimates don’t even approach the same stratosphere of the high-end methane leakage rate of 7.9 percent that Howarth and his colleagues manufactured to arrive at their conclusions. According to EPA, the actual rate is closer to 2.2 percent – a figure that recent studies suggest is far too high in its own right.
- According to Cornell professor Larry Cathles in a response paper published this month in the journal Climatic Letters: “While their low-end estimate of total leakages from well drilling through delivery (3.6%) is consistent with the EPA (2011) methane leakage rate of ~2.2% of production, and consistent with previous estimates in peer reviewed studies, their high end estimate of 7.9% is unreasonably large and misleading.” (page 2)
- Of course, the other problem is with the EPA estimates themselves. According to a report issued in August 2011 by the respected energy consulting firm IHS-CERA: “EPA’s [new] analysis relies on assumptions that are at odds with industry practice and with health and safety considerations at the well site. IHS CERA believes that EPA’s methodology for estimating these emissions lacks rigor and should not be used as a basis for analysis and decision making. … EPA derived the emissions factor from two slide presentations at Natural Gas STAR technology transfer workshops, one in 2004 and one in 2007. These two presentations primarily describe methane that was captured during “green” well completions, not methane emissions. EPA assumes that all methane captured during these green completions would have been emitted in all other completions. This assumption does not reflect industry practice.” (page 5)
Point #2: Just like last time, Howarth et al. assume that virtually all methane produced during the “flowback” phase of operations is simply vented into the atmosphere – not captured or burned.
The problem?
- For starters, it’s not even close to being correct. According to that IHS-CERA report: “Compounding this error is the assumption that all flowback methane is vented, when industry practice is to capture and market as much as possible, flaring much of the rest. Vented emissions of the magnitudes estimated by Howarth would be extremely dangerous and subject to ignition. The simple fact that fires are rare in all gas-producing areas suggests that this analysis grossly overestimates the quantities of methane that are leaking uncontrolled into the atmosphere at the well site.” (page 9-10)
- Even more problematic, Howarth et al. rely on flowback data they say comes from the Haynesville Shale to make their claims, simply extrapolating that out to account for and characterize all wells everywhere in the country. But according to Dr. Cathles of Cornell: “The numbers they use to represent fugitive emissions for the Haynesville Shale cannot be found in the references they cite.” (page 6) Added Cathles: “If a sales pipeline is not available, the gas captured by REC technologies could be easily be (and are) flared and the GHG footprint thereby minimized.” (page 3)
- Finally, Howarth et al. again cite EPA emissions data to support their contention that as much as 85 percent of the methane produced during the flowback stage is vented into the atmosphere. Interestingly, EPA’s actual estimates of vented methane come in at 49 percent – which, according to industry data, is itself wildly off the mark. According to Cathles: “Based on Howarth et al’s own references … we believe the losses during drill out and well completion for unconventional shale gas wells are not significantly greater than those cited by Howarth et al. for conventional gas wells. …This is supported by some of the examples cited by the EPA and Howarth et al. The Williams Corp (EPA 2007, p 14) shows, for example, that [greater than] 90 percent of the flowback gas is captured and some of the remainder flared (George 2011, p14).” (page 7)
Point #3: Howarth and his team refuse to acknowledge that their previous assumptions on lost-and-unaccounted for gas were incorrect, even in the face of a direct response/refutation by the U.S. Department of Energy.
The problem?
- As we highlighted in our initial rebuttal last year, Howarth, et al. estimate that between 1.4 percent and 3.6 percent of all natural gas produced over the life of a well leaks off into the atmosphere during the transmission process, a hypothesis that relies heavily on “lost and unaccounted for gas” (LUG) figures reported in a non peer-reviewed Texas trade magazine that went out of circulation in December 2010.
- CFR’s Michael Levi, reporting on a presentation produced last year by the National Energy Technology Laboratory (NETL), explains why that’s a bad idea: “The NETL documents don’t address the Howarth study explicitly, but if you flip to page 25, you’ll see a big part of the discrepancy explained. Some readers will recall that Howarth found a large fraction of produced gas from unconventional wells never made it to end users, assumed that all of that gas was vented as methane, and thus concluded that the global warming impacts were huge. As the NETL work explains, though, 62% of that gas isn’t lost at all – it’s ‘used to power equipment.’” (CFR blog, “Rebutting the Howarth Shale Gas Study,” May 20, 2011)
Given the lack of substantive response to these and several other important points highlighted in the Cathles paper, it’s understandable that Howarth and his team didn’t receive quite the same volume and intensity of press coverage this time around relative to their first effort back in April.
Unfortunately, though, much of the coverage they did get seeks to advance the simple narrative that this whole thing is just a friendly dust-up among faculty at Cornell – with Howarth and Ingraffea on one side, and Cathles, Larry Brown and Andrew Hunter on the other. This Bloomberg lede is typical of the approach: “Two groups of Cornell University researchers have split over the contribution to global warming by rising extraction of natural gas from shale beds through a process known as fracking.” (Bloomberg, Jan. 20, 2012)
Of course, the reality of the situation is quite a bit different. Actually, it’s Howarth and Ingraffea on one side, and the rest of the intelligent world on the other. In the nine months since their initial paper was published, detailed responses have been compiled and released by no fewer than a dozen separate academic, government and non-governmental institutions (a quick list of those is available here and here). In a study commissioned by the Sierra Club, one researcher even went so far as to call the Howarth data biased: “We don’t think they’re using credible data and some of the assumptions they’re making are biased. And the comparison they make at the end, my biggest problem, is wrong.”
Of course, over on Planet Ithaca, not only don’t any of these criticisms hold merit – according to the Cornell research team, they don’t even exist. In an online chat hosted by the Syracuse Post-Standard in September, Prof. Ingraffea provided the following answer when asked how he’s dealing with the all the controversy that his paper has engendered: “We have not received any of what we would consider intense peer criticism.” Which we guess is true. Just so long as you continue to believe it.
ICYMI: Criticism of Cornell shale paper “leaks” out from some unlikely places
Posted April 13th, 2011 by Dave McCabe, atmospheric scientist
Keep Reading »
