[ISN] Surviving Rita: A tale of two data centers

[InfoSec News]

http://www.computerworld.com/securitytopics/security/recovery/story/0,10801,104940,00.html

By Todd R. Weiss 
SEPTEMBER 24, 2005
COMPUTERWORLD

For the second hurricane in a row, New Orleans-based Web hosting 
vendor and Internet domain name registrar DirectNIC managed to stay up 
and running overnight as Hurricane Rita pounded the Gulf Coast before 
coming ashore early this morning. 

Sigmund Solares, CEO of Intercosmos Media Group Inc., which owns 
DirectNIC, rode out Hurricane Rita in New Orleans last night and was 
busy assessing damage this morning. 

Solares said his company, located on the 10th floor of a downtown New 
Orleans office building, was still dealing with broken windows and 
other damage left from Katrina, which plowed through the area Aug. 29. 
During that storm - and despite massive flooding in much of New 
Orleans after -- DirectNIC's operations continued uninterrupted 
because it had an emergency generator and an adequate fuel supply, he 
said. 

DirectNIC has approximately 800 hosted servers. 

"We definitely got hurt by Katrina and we still have a ways to go [to 
get completely cleaned up] but we remained open the whole time," 
Solares said. 

Even as Rita's strong winds brushed by last night, ripping tin pieces 
from neighboring structures and hurtling them at DirectNIC's building, 
Solares said he and his four team members were much less worried than 
when Katrina struck. "It's still windy outside," he said this morning. 
"It's still gusty. [But] right now, we are feeling much better than at 
any time during Katrina." 

Solares said he got a good night's sleep inside an internal office in 
the building, where he was able to get away from the howling winds 
outside. The building continued to have electricity until 7:59 a.m. - 
more than four hours after the storm struck near the Louisiana-Texas 
border. The power went out then, and battery backups kicked on 
instantaneously, followed by a backup emergency generator, he said. 
The outage only lasted 11 minutes, however, and the company's servers 
remained up and running. 

Looking outside, Solares said damage to neighboring buildings doesn't 
look as bad as when Katrina hit. Floodwaters didn't reach his building 
during the last storm, nor did they do so this time. "With Katrina, 
when the levees broke, I could see the flooding a half a block away," 
he said. "I'm looking where there was flooding [before] and I don't 
see any water there right now."

In other parts of the city, particularly the Ninth Ward, floodwaters 
cascaded over newly-repaired levee walls yesterday, causing more 
problems in low-lying areas. 

During Katrina, and again last night, DirectNIC's employees had to 
build a makeshift levee of their own. Rainwater began entering through 
the ceiling from broken windows on the floor above, Solares said, 
forcing him and his staff to use mops, a wet-dry vacuum machine and 
bundles of promotional T-shirts to create what he called the company's 
own "levee system" to keep water away from servers and other 
electronic gear. Those same methods came in handy again last night, he 
said, since building repairs have not yet been made. Before Rita drew 
near, Solares and his crew installed plywood over the open windows and 
used sealant and tarps, hoping to keep out as much rain as possible. 
The makeshift repairs helped, he said. 

Earlier this week, as Rita swirled in the Gulf of Mexico as a 
dangerous category 5 storm, Solares called a competing Web hosting 
vendor in Houston, EV1Servers.net, and offered help in case Rita hit 
Houston dead-on. 

"For us, there's a lot we can do," Solares said. "It's not much effort 
for us to help people get back up and running." DirectNIC has extra 
space and capacity for servers and can make that space available to 
others in an emergency, he said. 

"We're lucky that we've always saved for a rainy day and that we're 
able to help our employees and try to help other people, even though 
it's a very tough time for us overall," Solares said. "It's also good 
for morale for our employees to see us helping other people in need." 

In Houston, Robert Marsh, the CEO of EV1Servers.net, said early today 
that because Hurricane Rita tracked farther toward the east, away from 
Houston, the city was spared the storm's worst effects (see "Update: 
As Rita drew near, Web hosting vendor prepared" [1]). The highest wind 
gusts were in the 50-mph-to-60-mph range, far lower than the 120 mph 
winds expected at one point. By this morning, only an inch of rain had 
fallen in the area where EV1's data centers are located, he said. 

"In this instance, we were very fortunate not to have received the 
worst," he said. Overnight, the company switched over to its emergency 
generators after some nearby power transformers blew up. EV1 plans to 
return to utility power sometime today after stable power flow is 
maintained for at least five hours, he said. 

About 25 EV1 employees stayed overnight inside the company's offices 
and two data centers to maintain services for customers. The company 
had also sent seven Web technicians to a hotel in Wichita, Kan. to 
remotely handle customer trouble tickets had its main data centers 
been knocked out by the storm. 

"Our trouble ticket load is essentially back to normal at this time," 
Marsh said today. "If the storm continues to clear, we hope to bring 
our people back from Wichita as early as Sunday night."

[1] http://www.computerworld.com/securitytopics/security/recovery/story/0,10801,104892,00.html