This week, the online publication InsideClimate News posted an article that attacks a new study led by Prof. Donald Siegel of Syracuse University, claiming that it should be “called into question because of its methodology and some undisclosed ties” to a natural gas producer.

Many scientists, engineers and environmental regulators have found that hydraulic fracturing is safe and the risks associated with it are manageable. But notwithstanding this mountain of evidence and testimonials, activists routinely try to advance this narrative that the only research out there that exists even vaguely supportive of responsible development must have come from the industry itself. And if someone from the industry contributed even a single data point or well sample to the project, then the whole enterprise must be dismissed out of the hand, in its entirety.

As EID explained when the Siegel study came out, the paper pretty much directly debunks the two most controversial papers (2011 and 2013) that came from Robert Jackson and his team at Duke Univ. (Prof. Jackson is now at Stanford). As readers will remember, those papers tried to establish a causal link between the existence of methane in water wells in Pennsylvania and shale development nearby, arguing that the closer a water well is to a gas well, the more likely it is to be “contaminated” by methane.

Of course, EID has over time documented a number of serious methodological flaws in the Duke studies, which include the use of a picayune sample size, no random sampling, no baseline data, and the fact that researchers found high levels of methane in lots of water wells residing nowhere near natural gas sites. They also focused on sampling those wells where they knew ahead of time they were most likely to find issues — the exception rather than the rule.

The new Siegel paper specifically addresses the limitations of those Duke studies: it uses a dataset hundreds of times larger, establishes pre-drill baselines, and samples statistically almost the entire population of domestic wells in the study area, not just a very small fraction of them.  To be sure, the study finds lots of methane in water wells, but concludes: “there is no significant correlation between dissolved methane concentrations in groundwater and proximity to nearby oil/gas wells.”

Against that backdrop, let’s take a closer look at this InsideClimate attack piece:

InsideClimate claim: “the study authors failed to divulge the scope of their ties to Chesapeake, including fees the company paid to Siegel to carry out his research. One of the paper’s four co-authors, Bert Smith, worked for Chesapeake during some of the period when the study took place, which also wasn’t disclosed. Smith works for the company today. The paper only acknowledges that Chesapeake provided the dataset.”

FACT:  For those who read the paper carefully, it should be clear to most that the data used in this study came from Chesapeake’s own internal records, which required some form of payment to access. It should have also been fairly obvious that Prof. Siegel’s coauthors were employees of consulting firms, and that those folks typically do not work for free.

Bert Smith was not an employee of Chesapeake when he contributed to the paper; he worked for a consulting firm. Prof. Siegel, for his part, did not receive a grant from Chesapeake either, but he did tell us that he received the equivalent of about one month of his salary to provide advice on how to conduct the study properly and take the lead on writing the paper.

But all this is irrelevant because reviewers of the paper and the Journal’s editorial staff found no fault with the initial disclosure. Still, Prof. Siegel and colleagues revised their disclosure to meet the “concerns” and the Journal accepted the revision.  Further, the enormous data set that was used to develop this paper (note how silent the activists continue to be about this) stands on its own.

InsideClimate claim: “Further, the contractors used the so-called inverted bottle method to collect the samples, which entails placing an upside-down bottle in a bucket of water to collect the escaping gas. Jackson said he does not use the inverted bottle method because it doesn’t produce reliable results. The United States Geological Survey and some other major water testing labs do not use the technique to measure methane.” 

FACT: If the sampling process is managed in a sloppy way, then it’s certainly true that gas can bubble out.  But given the regulatory controls that were applied as part of this analysis, Prof. Siegel tells us that “sloppy sampling is highly unlikely.”  And even if some very high concentration samples degassed during sampling, they would still be close to equilibrium values for methane dissolved in water.  This is above Pennsylvania Department of Environmental Protection’s (DEP) threshold for concerns regarding methane in drinking water, which are less than 10 mgL. And if there were error, it would be systematic. High concentrations would still be high and low concentrations would be low.

Further, even if a few samples did turn out to be sloppy, the sheer number of samples would make the ones in error simple outliers. In other words, in order for InsideClimate and Prof. Jackson to be able to discredit the Siegel study’s methodology, a majority of the 10,000 samples would have had to have been sloppy.  Obviously, given the regulatory protocols that were in place, this is completely implausible.

InsideClimate claim: “Chesapeake hired contractors to sample the water at the tap. As a result, some of the water had already undergone treatment on its way from well to faucet. Treatment and running the water adds “noise” to the readings and affects the amount of methane in a sample, Jackson said.”

FACT:  Regulatory protocol demands that samples of domestic water wells be collected at the tap after flushing the system of stagnant water.  The purpose of Siegel’s study was to use this very same protocol to determine the extent of natural variability using the method that the state uses to flag potential gas well problems. Those who think hydraulic fracturing is harmful to water can’t have it both ways— they can’t argue that methane coming out of taps near gas wells could be from fugitive gas releases, and then say that sampling this way is improper if no gas is found or if high values are found far away from gas wells.

InsideClimate claim: “The studies (by Jackson’s group) suggested the Marcellus methane ended up in well water because of the wells’ faulty metal casings that allowed the methane to seep out into aquifers as natural gas was pumped to the surface. The research said the leaks could also be linked to poor concrete construction inside gas wells.”

FACT: Sampling systematically to find methane released from gas wells where it clearly had happened previously, and then arguing this exception (based on well known data) may be the rule, is not science. It proves the obvious point that where fugitive gas occurs, domestic wells will get more gas.   As the Science Insider reported:

“Siegel believes the PNAS studies, led by Robert Jackson … painted a more worrisome picture because their small sample set was skewed toward locations with known well-casing issues.”

As we’ve noted before, well failure rates are exceedingly rare.  The Associated Press recently completed an investigation of water contamination and well integrity, and using data from the Pa. DEP, found a well failure rate of only about one-third of one percent (0.33 percent) in Pennsylvania.  Further, a 2011 study by the Ground Water Protection Council (GWPC) found a well failure rate of only 0.03 percent in Ohio and only about 0.01 percent in Texas.  There are others, too. And, those numbers pertain to operating situations that predate many of the new casing regulations that have gone into effect in those states in recent years.  Past practices are not practices of today.

Compare the Duke approach to the Siegel study, which also looks at methane in proximity to existing gas wells but tested water wells in over 10,000 samples, statistically the entire population of domestic wells.  As Science Insider also reported, “Siegel doesn’t deny that there have been problems with a few wells with poorly engineered steel casings or cracked and degraded cement walls designed to keep the boreholes from leaking.”  But, it reports, “his study shows that it is an exceedingly rare issue.”

A Tale of Two Studies

As mentioned, the Duke study relied on a tiny data set (less than 150 samples) and systematically sampled near known gas releases.  The Duke authors also found thermogenic methane in nearly every well they sampled – even in wells nowhere near natural gas development; yet they posed a potential serious problem still. Peer-reviewed comments took issue with the Duke team’s conclusions, saying there was a “lack of data” to support their conclusions, and that the report “misrepresents potential risks” of shale development.

Two years later, the same team published another report that suffers from the same problems.  They found methane in 82 percent of the water wells they sampled. More than fifty of those wells – among the 141 total homes sampled – were nowhere near natural gas wells.  They also refrained from random sampling.  Finally, they suggest thermogenic methane concentrations may be linked to Marcellus wells, but pre-drill testing in the exact same part of the state by the federal U.S. Geological Survey report directly contradicts that thesis.  Also, a recent study by led by Fred Baldassare from Echelon Applied Geochemistry Consulting analyzed groundwater in northeastern Pennsylvania and found large amounts of thermogenic gas: 88 percent of the 67 water wells tested had some presence of thermogenic gas, and none of those sampled showed the presence of Marcellus gas.

Compare Duke’s fewer than 150 samples to the Siegel study, which draws on data from thousands of sites and corroborates findings from the USGS and many others pointing to naturally occurring methane in Pa. water wells, and the conclusion on this doesn’t need to be complicated. After all: the strength of your study derives from the strength of your data set. And does anyone actually doubt who’s got the better one of those?