The silhouettes of people are seen watching the sunset over the Schuylkill River in Philadelphia, May 8, 2015. Philadelphia. 
Photo by Victor J. Blue/Bloomberg/Getty

Is Philadelphia testing its drinking water correctly?

Philadelphia’s water passed its last round of lead testing with flying colors, but documents obtained by NBC News show those results may offer a misleading picture of the lead flowing from Philly taps.

A memo obtained by NBC News through a records request shows that in 2014 in a city with up to 50,000 homes with lead pipes, Philadelphia based its clean bill of health on sampling just 34 homes with lead pipes — fewer than required by law — while instead testing far more homes without lead pipes. After inquiries from NBC News, the Pennsylvania Department of Environmental Protection said it is now reviewing Philadelphia’s data.

Experts say the choice of houses, coupled with other flaws in the city’s testing methods, may be offering Philly’s residents false comfort about the quality of their water - and are a symptom of nationwide testing failures that may have masked lead problems in cities across the country, including Flint. Compared to some other U.S. utilities, Philadelphia Water has a reputation for being well-run and resourced.

“I have the utmost respect for the folks in Philadelphia, and if this is what’s going on in Philadelphia I shudder to think what’s going on in the rest of the country,” said Dr. Marc Edwards, a Virginia Tech researcher who was key to exposing the Flint lead crisis.

The Rachel Maddow Show, 2/17/16, 9:52 PM ET

Flint mayor ready to replace pipes as state, governor delay

Mayor Karen Weaver of Flint, Michigan talks with Rachel Maddow about how she intends to commence with replacing the city’s water pipes as early as next week as Governor Rick Snyder and the state legislature continues to drag their feet on funding and…

State and local officials are struggling to follow a federal regulation known as the Lead and Copper Rule that’s designed to gauge whether a utility is properly treating its water to prevent corrosion of pipes that would leach heavy metals like lead into the water.

If 90 percent of homes tested register below the “action level” of 15 parts per billion of lead, the system is considered compliant. But testers are supposed to focus on high-risk homes, meaning older structures that have lead pipes or lead service lines in order to get a worst-case picture of the system.

Despite assurances by Flint and Sebring, Ohio, utilities that they were testing high-risk homes, neither was. Now officials in both towns are scrambling to address lead issues overlooked for months or years.

In a city like Philadelphia, which has old housing stock and thousands of lead pipes, the rule requires at least half the homes tested be “high risk.”

But documents obtained by NBC News show that Philadelphia Water was missing many high-risk homes, and getting test results from neighborhoods with relatively new, and therefore lower risk, homes. The percentage of high-risk homes tested had also dropped over time. The results show that the homes picked may determine whether the city seems to have safe water or seems to have a problem.

In 2002, 54 of the homes tested in Philadelphia were highest-risk “Tier 1” homes, meaning they had lead pipes or copper pipes with lead solder laid just before the lead solder ban in 1986. In 2002, the city only tested a total of 63 homes, and only nine were low-risk.

Philadelphia’s all-important 90th percentile — the result that triggers its overall grade — came in at 13 parts per billion, just two points under the “action level” that requires utilities warn customers of the risk. Five lead-line houses that year tested high, as did one low-risk copper-line home.

By the following round of testing, three years later, most of the high-lead homes had dropped out of the testing pool. Since then, as the number of high-risk homes in the test group dwindled while the overall pool of tested homes grew, the city’s results improved.

In 2014, Philadelphia tested just 34 homes with lead lines, and eight with high-risk lead solder. Those 42 high-risk homes made up less than a third of the 134 homes tested. That year, the city’s 90th percentile had 5 ppb of lead in its water, well under the 15 ppb action level. Eight results tested high that year — three lead-line homes, and five lower-risk copper pipe homes.

The trend of choosing low-risk homes is combined with sampling instructions for test homes, first revealed in documents obtained by the Guardian, that may also skew the results downward.

Residents are told to remove their tap aerator and run their water the night before the test to flush out the line, known as “pre-flushing,” a practice that EPA water experts say can produce misleading results and has been used in other cities, like Flint.

RELATED: Parents of ‘tragic’ 2-year-old with lead poisoning sue Flint

“This clears particulate lead out of the plumbing and biases the results low,” wrote EPA scientist Miguel Del Toral in a February 2015 email about Flint’s testing obtained by researchers at Virginia Tech. This practice “provides false reassurance about the true lead levels in the water,” he added.

Asked if Philadelphia’s method of choosing homes and “pre-flushing” could skew results, EPA water expert Mike Schock said, “In my technical opinion, yes it would.”

Philadelphia counters that its testing methods paint an accurate picture of a city where high lead is an anomaly, not the rule.

“It’s not clear that the way we’re taking samples is hiding problems,” said Gary Burlingame, director of Philadelphia Water’s Bureau of Laboratory Services. “Could we be missing homes out there that have higher lead levels than others? Of course.”

Utilities across the U.S. have told regulators about the challenges of meeting the federal rule’s demands. Cities must solicit volunteers for their water studies, and then ask the volunteers to run the tests themselves. In its memo to regulators, Philadelphia Water said it sent over 8,000 letters to customers, and only 134 had followed through the process.

“People aren’t responding,” said Burlingame, and often don’t continue testing from one year to the next. “Lead in water just hasn’t been something that’s been a national issue.”

The state Department of Environmental Protection accepted Philadelphia’s results, and the city’s argument that it was not able to get more high-risk homes. This despite the state agency’s own guidance that, in accord with the federal rule that at least half of the homes tested in cities like Philly, with old homes and lots of lead pipes, should be high-risk.

RELATED: Hillary Clinton: There are too many Flints

The state also has guidance that any homes tested beyond the required number be high risk so as not to “dilute” the testing pool. Philadelphia was only required to test 50 homes in total, but tested 134 — most of them low-risk. Burlingame said the utility tested the extra homes because it believed a larger pool would give a more accurate picture of the system.

He added that flushing the line was to clear any hot water from the tap. The federal rule is designed to test cold water, he said.

The DEP said that while half of homes tested should be high risk, sometimes a utility is not able to find enough high-risk homes. However, the agency said it is now reviewing the data submitted by Philadelphia.

The agency also said that Philadelphia had not “diluted” its pool in 2014, despite the extra homes in the sample, because more low-risk homes had tested high than high-risk homes.

“The tier 3 [low risk] sample actually skewed the overall sample level higher, while still registering below action levels,” the DEP said in a statement. Burlingame also said the city has seen problems in lower-risk homes in the past.

In addition, both the DEP and the EPA have issued explicit guidance telling utilities not to remove aerators when testing. But the DEP said it can’t make utilities comply.

“Both state and federal guidelines do recommend that aerators not be removed,” said a DEP spokesperson. “However, states cannot enforce recommendations or guidance.”

Flaws in testing have not stopped officials across the country from trying to calm the fears of residents in the wake of Flint. Earlier this month, based on testing results from Philadelphia and more than 150 other state water utilities that came in under the “action level,” the state DEP told Pennsylvanians that “drinking water is not the source” of elevated lead levels found in some of the state’s children.

DEP Secretary John Quigley sent a clear message to the people of the state: “DEP has regulations and programs in place to monitor lead levels in drinking water, and they are working.”

This article originally appeared on NBCNews.com.

Flint, Philadelphia and Water

Is Philadelphia testing its drinking water correctly?