[Infowarrior] - Massive Flooding Damages Several NYC Data Centers

[Richard Forno]

(Just getting fully back online now.  --rick)

Massive Flooding Damages Several NYC Data Centers
By: Rich Miller
October 30th, 2012

http://www.datacenterknowledge.com/archives/2012/10/30/major-flooding-nyc-data-centers/

Flooding from Hurricane Sandy has hobbled two data center buildings in Lower Manhattan, taking out diesel fuel pumps used to refuel generators.A third building at 121 Varick is also reported to be without power. There were also reports of outages for some tenants at a major data hub at 111 8th Avenue, and many other New York area facilities were running on generator power amid widespread utility outages.

Both Internap and Peer 1 are struggling to continue operations at 75 Broad Street after basement-level flooding disabled critical diesel fuel pumps, leaving the providers no way to refuel generators on mezzanine floors.

Customers of Datagram were knocked offline Monday evening as water flooded the basement of its building at 33 Whitehall, knocking out high-traffic sites including Gawker, Gizmodo, Buzzfeed and Mediate.

Both 75 Broad and 33 Whitehall were located in the “Zone A” flood zone, which was inundated by a storm surge of more than 13 feet as Sandy’s landfall in New Jersey coincided with high tide in lower  Manhattan.

Meanwhile, Atlantic Metro reports that several of its data centers are offline, including its facility at 121 Varick, which has “significant flooding” in its generator room.

There were also multiple reports of downtime for some tenants at 111 8th Avenue, a major communications hub which is owned by Google. Some reports attributed the outages to network problems, while others said that an electrical panel had failed while parts of the building were switching to generator power.

Meanwhile, several data centers in northern New Jersey were reported to be running on generator power, as PSE&G and other utilities suffered widespread outages.

Datagram

Datagram hosts a number of high-profile blogs, including the Gawker network, which remains offline Tuesday morning. “Gawker sites down after power cut off at Datagram, our data center down on Whitehall St,” tweeted Gawker publisher Nick Denton. “Backup power didn’t kick in fast enough.”

In fact, the problems at 33 Whitehall went beyond the speed of its power response. Other tenants reported that the basement had been flooded, knocking out diesel fuel pumps that support generators.

Buzzfeed said it was able to work with content delivery provider Akamai to restore a static version of its web site. It posted a short story about the problems experienced by Datagram, including a text sent to Buzzfeed from a Datagram official. “Basement flooded, fuel pump off line – we got people working on it now. 5 feet of water now,” the official wrote.
It wasn’t clear when Datagram would be able to restore service.

75 Broad

Diesel pumps were also the focus of problems at 75 Broad, where basement levels experienced major flooding that forced Internap and Peer 1 to shut down their operations at the facility.

“Around 9 PM the storm surge, combined with high tide, caused flooding to many portions of lower Manhattan, including in and around our data center at 75 Broad,” Internap said in a  blog post. “At the peak there was three feet of flood water in the lobby at the site, causing both basement floors to be inaccessible. As a result of the flooding, both our redundant fuel pumps and our generator fuel tank were compromised and shut down. The system continued to run until all fuel within the secondary feeder tanks were exhausted and our facility lost power.”

The Internap facility lost power at about 11:45 a.m. this morning.

“We are working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation,” Internap said. “It is unclear how long it will take ConEd to restore utility power to the site, but we are preparing for the possibility of remaining on generator power for many days.”

UPDATE: “We have coordinated fuel deliveries and pumps, as well as engineers to fabricate pipe to bring the fuel directly to the generators on the mezzanine level,” Internap reported this afternoon. “There is currently no ETA available for the completion of these activities, but we are working through all options to bring the UPS infrastructure back online as soon as possible. We will continue to work toward resolution and
provide status updates with any new information.”

After attempting to arrange temporary diesel fuel storage and pumps, Peer 1 Hosting said early Monday that it will “implement a controlled shutdown of NY Data Center at 10:45 ET.” But as of 12:30 p.m., the site was still online, and the company said it was awaiting a diesel delivery.

“A fuel delivery is en route to the facility but due to the various street closures and general road conditions caused by Hurricane Sandy, (and) the expected ETA of the fuel truck is approximately 1 hour, Peer 1 said on its support forum. “If the fuel delivery is not able to be made before 2:00PM EST, PEER 1 will need to initiate a suspension of all services at our affected facility beginning approximately at 2:15PM EST.”

Peer 1 has not provided any updates since the 2:15 deadline.

Atlantic Metro / 121 Varick

On Tuesday morning Atlantic Metro reported that it was experiencing “widespread network and facility power outages due to Sandy and its flooding.” The company was experiencing significant problems at multiple data centers:

LGA1 (325 Hudson Street) – “We are working with local building crew to clear the area of the fuel pumps needed for Diesel generator power. Replacement fuel pump estimated to be received (by 9:30 p.m. Tuesday). Work will commence to restore service at that time.” Atlantic Metro says repairs at LGA1 will address a connectivity problem at LGA6.

LGA4 (121 Varick Street) – “ There is significant flooding in the generator room of this IDC. We are working with FDNY to safely investigate the area. (The) site is suffering from a damaged ATS. Crews will be called in first thing tomorrow morning to begin repair efforts.”

111 8th Avenue

There were multiple reports on Twitter of outages for some tenants at 111 8th Avenue, one of Manhattan’s largest buildings and a key hub for Internet traffic. Companies affected included Equinix, Voxel/Internap and XO Communications.

“NY9 experienced a failed generator that impacted service to several customers,” Equinix said on it blog, referring to its facility at 111 8th Avenue. “We made repairs and service was returned this morning. The site remains on generator power.”

111 Eighth Avenue is among the world’s most wired buildings. It was originally built as the Port Authority Commerce Building in 1932, and was redeveloped for telecom use by Taconic in the late 90s.

25 Broadway (Telehouse)

Telehouse reported that its facility at 25 Broadway in lower Manhattan has remained online throughout Sandy and its aftermath, as have its data center in Chelsea and at the Staten Island Teleport. The colocation provider said it switched to generator power proactively ahead of Con Edison’s utility power shutdown. “Since that time, these facilities have remained operational without any critical system (power, cooling) outages,” Telehouse reported. ” These facilities will remain fully staffed and off-grid (independent power generation mode), with periodic refueling until utility power is restored and stable.

“During this period, both Manhattan sites (25 Broadway, Chelsea) were operationally unaffected by local flooding,” the company added. “Building engineers and equipment, at both Manhattan facilities, are engaged in mitigating this threat. The Teleport facility, on Staten Island, had no flood threat during this time.”

32 Avenue of the Americas (CoreSite)

CoreSite reported that its data centers all remained online throughout Sandy, although its data center at 32 Avenue of the Americas in New York switched over to generator power Monday evening. “That transition occurred smoothly according to established procedures,” CoreSite said in a statment. “CoreSite is checking regularly with Con Edison as to when normal power is expected to be restored, but remains confident in its ability to continue running on its emergency generators as long as necessary given that fuel deliveries have been secured and remain on schedule.”

“Our thoughts go out to the people and communities affected by the devastation of this storm,” said CoreSite President and CEO Tom Ray. “Amidst this much bigger picture, we are striving to ensure our customers have timely and accurate information as to the current state of site operations and understand how to reach us.”

Telx

Colocation and interconnection specialist Telx is the largest service provider at 60 Hudson Street, one of the the leading carrier hotels in Manhattan. Telx says 60 Hudson lost power Monday at 7:15 p.m. but is operating on generator power, and has enough fuel to operate through Wednesday.

Telx also has facilities at 111 8th Avenue and says it is operating fine on generator power.

We will continue to update as we get more news. You can email us at news at datacenterknowledge.com with news and tips.
---
Just because i'm near the punchbowl doesn't mean I'm also drinking from it.