[Infowarrior] - Comedy of Errors Led to False ‘Water-Pump Hack’ Report

[Richard Forno]

(c/o DOD, no not that one!)

Exclusive: Comedy of Errors Led to False ‘Water-Pump Hack’ Report

	• By Kim Zetter  
	• November 30, 2011  | 
	• 5:54 pm  | 
	• Categories: Cybersecurity, Hacks and Cracks

http://www.wired.com/threatlevel/2011/11/water-pump-hack-mystery-solved/

It was the broken water pump heard ’round the world.

Cyberwar watchers took notice this month when a leaked intelligence memo claimed Russian hackers had remotely destroyed a water pump at an Illinois utility. The report spawned dozens of sensational stories characterizing it as the first-ever reported destruction of U.S. infrastructure by a hacker. Some described it as America’s very own Stuxnet attack.

Except, it turns out, it wasn’t. Within a week of the report’s release, DHS bluntly contradicted the memo, saying that it could find no evidence that a hack occurred. In truth, the water pump simply burned out, as pumps are wont to do, and a government-funded intelligence center incorrectly linked the failure to an internet connection from a Russian IP address months earlier.

Now, in an exclusive interview with Threat Level, the contractor behind that Russian IP address says a single phone call could have prevented the string of errors that led to the dramatic false alarm.

“I could have straightened it up with just one phone call, and this would all have been defused,” said Jim Mimlitz, founder and owner of Navionics Research, who helped set up the utility’s control system. ”They assumed Mimlitz would never ever have been in Russia. They shouldn’t have assumed that.”

Mimlitz’s small integrator company helped set up the Supervisory Control and Data Acquisition system (SCADA) used by the Curran Gardner Public Water District outside of Springfield, Illinois, and provided occasional support to the district. His company specializes in SCADA systems, which are used to control and monitor infrastructure and manufacturing equipment.

Mimlitz says last June, he and his family were on vacation in Russia when someone from Curran Gardner called his cell phone seeking advice on a matter and asked Mimlitz to remotely examine some data-history charts stored on the SCADA computer.

Mimlitz, who didn’t mention to Curran Gardner that he was on vacation in Russia, used his credentials to remotely log in to the system and check the data. He also logged in during a layover in Germany, using his mobile phone.

“I wasn’t manipulating the system or making any changes or turning anything on or off,” Mimlitz told Threat Level.

But five months later, when a water pump failed, that Russian IP address became the lead character in a 21st-century version of a Red Scare movie.

Jim Mimlitz at the airport in Frankfurt, Germany, during a layover last June on his way to Russia. Courtesy of Jim Mimlitz.

On Nov. 8, a water district employee investigating the pump failure called in a contract computer repairman to check it out. The repairman examined the logs on the SCADA system and saw the Russian IP address connecting to the system in June. Mimlitz’s username appeared in the logs next to the IP address.

The water district passed the information to the Environmental Protection Agency, which governs rural water systems. “Why we did that, I think it was just out of an abundance of caution,” says Don Craven, a water district trustee. “If we had a problem we would have to report it to EPA eventually.”

But from there, the information made its way to the Illinois Statewide Terrorism and Intelligence Center, a so-called fusion center composed of Illinois State Police and representatives from the FBI, DHS and other government agencies.

Even though Mimlitz’s username was connected to the Russian IP address in the SCADA log, no one from the fusion center bothered to call him to ask if he had logged in to the system from Russia. Instead, the center released a report on Nov. 10 titled “Public Water District Cyber Intrusion” that connected the broken water pump to the Russian log-in five months earlier, inexplicably stating that the intruder from Russia had turned the SCADA system on and off, causing the pump to burn out.

“And at that point … all hell broke loose,” Craven said.

Whoever wrote the fusion center report assumed that someone had hacked Mimlitz’s computer and stolen his credentials in order to use them to hack into Curran Gardner’s SCADA system and sabotage the water pump. It’s not clear whether it was the computer repairman or the fusion center that first jumped to this conclusion.

A spokeswoman for the Illinois State Police, which is responsible for the fusion center, pointed the finger at local representatives of DHS, FBI and other agencies who are responsible for compiling information that gets released by the fusion center.

“We did not create the report,” said spokeswoman Monique Bond. “The report is created by a number of agencies, including the Department of Homeland Security, and we basically are just the facilitator of the report. It doesn’t originate from the [fusion center] but is distributed by the [fusion center].”

But DHS is pointing the finger back at the fusion center, saying if the report had been DHS-approved, six different offices would have had to sign off on it.

“Because this was an Illinois [fusion center] product, it did not undergo such a review,” a DHS official said.

The report was released on a mailing list that goes to emergency management personnel and others, and found its way to Joe Weiss, managing partner of Applied Control Solutions, who wrote a blog post about it and provided information from the document to reporters.

The subsequent media blitz identified the intrusion as the first real hack attack against a SCADA system in the U.S., something that Weiss and others in the security industry have been predicting would happen for years.

The hack was news to Mimlitz.

He put two and two together, after glancing through his phone records, and realized the Russian “hacker” the stories were referring to was him.

Teams from the FBI and DHS’s Industrial Control Systems-Cyber Emergency Response Team (ICS-CERT) subsequently arrived in Illinois to investigate the intrusion and quickly determined, after speaking with Mimlitz and examining the logs, that the fusion center report was wrong and should never have been released.

“I worked real close with the FBI and was on speakerphone with the fly-in team from CERT, and all of them were a really sharp bunch and very professional,” Mimlitz said.

DHS investigators also quickly determined that the failed pump was not the result of a hack attack at all.

“The system has a lot of logging capability,” Mimlitz said. “It logs everything. All of the logs showed that the pump failed for some electrical-mechanical reason. But it did not have anything to do with the SCADA system.”

Mimlitz said there was also nothing in the logs to indicate that the SCADA system had been turned on and off.

He cleared up another mystery in the fusion report as well. The report indicated that for two to three months prior to the pump failure, operators at Curran Gardner had noticed “glitches” in their remote access system, suggesting the glitches were related to the suspected cyber intrusion.

But Mimlitz said the remote access system was old and had been experiencing problems ever since it was modified by another contractor.

“They had made some modifications about a year ago that was creating problems logging in,” he said. “It was an old computer … and they had made network modifications that I don’t think were done correctly. I think that’s why they were seeing problems.”

Joe Weiss says he’s shocked that a report like this was put out without any of the information in it being investigated and corroborated first.

“If you can’t trust the information coming from a fusion center, what is the purpose of having the fusion center sending anything out? That’s common sense,” he said. “When you read what’s in that [report] that is a really, really scary letter. How could DHS not have put something out saying they got this [information but] it’s preliminary?”

Asked if the fusion center is investigating how information that was uncorroborated and was based on false assumptions got into a distributed report, spokeswoman Bond said an investigation of that sort is the responsibility of DHS and the other agencies who compiled the report. The center’s focus, she said, was on how Weiss received a copy of the report that he should never have received.

“We’re very concerned about the leak of controlled information,” Bond said. “Our internal review is looking at how did this information get passed along, confidential or controlled information, get disseminated and put into the hands of users that are not approved to receive that information. That’s number one.”

Additional reporting by Ryan Voyles in Illinois.

Kim Zetter is a senior reporter at Wired covering cybercrime, privacy, security and civil liberties.
Follow @KimZetter and @ThreatLevel on Twitter.


---
Just because i'm near the punchbowl doesn't mean I'm also drinking from it.