[ISN] Vulnerable to risk: Other resorts susceptible to a similar outage

[InfoSec News]

http://www.lasvegassun.com/sunbin/stories/sun/2004/apr/15/516694304.html

April 15, 2004 
By Liz Benston 
<benston at lasvegassun.com>
LAS VEGAS SUN

The power outage that shut down the Bellagio and caused the evacuation
of thousands of employees and customers could happen at any resort on
the Las Vegas Strip and at any time, Clark County's top building
inspector said Wednesday.

That's primarily because the power failure occurred on the property
and with Bellagio equipment rather than on an outer line supplied by
Nevada Power Co., Clark County Department of Development Services
Building Division Director Ron Lynn said.

Some newer resorts such as the Bellagio own several power generators
on site that can create instantaneous backup power to run their
resorts should Nevada Power lines fail, he said.

If Nevada Power's lines to the Bellagio had failed, the property's
generators would have kicked on, flooding the property with the same
power load in about the time it takes to flick a switch, he said.

"(Resort guests) wouldn't have known the difference," he said. "Maybe
the lights would have flickered and that's it."

But that's not what happened at the Bellagio Sunday, he said. The
failure occurred internally on resort-owned power lines, meaning the
power carried from Nevada Power lines had nowhere to go and generators
couldn't pick up the slack, he said. The generators are set up to go
on if power doesn't come on to the property.

An internal power line failure could mean bad news for any major
casino, resort officials say.

"The same kind of initial outage could happen to anyone," Harrah's
Entertainment Inc. spokesman Gary Thompson said.

What happens next could be more crucial, however, he said.

Harrah's Rio hotel-casino has taken the rare step of building its own
combustion turbine power plant that is designed to provide "more power
than needed to get the resort up and operating very quickly" should
the resort's main power line fail, Thompson said. With the power plant
and following proper procedures, the resort would be able to power up
the resort in a few minutes and wouldn't have to rely on emergency
power and evacuate guests in the process like the Bellagio did, he
said.

The 4.92 megawatt plant, designed to provide at least 40 percent of
the resort's power load, will be up and running in the next several
weeks and is expected to cost the company about $7 million.

Before adding the plant, the resort had a primary power source and an
emergency power source. The emergency source is a separate cable line
that is required of all resorts and provides minimal lighting so that
guests can evacuate the property and powers safety features such as
fire alarm system, smoke detectors and heat detectors.

The Bellagio also has a main power line and an emergency, or "safety"  
backup line to maintain minimal power for safety purposes. When some
cables in the main line shorted, the property shut off power in all
the main cables to repair the damaged ones and relied on its separate
emergency line, which is fed by a separate line provided by Nevada
Power, Lynn said.

The Bellagio's power system meets code requirements and the resort
followed proper safety procedures by shutting down its primary power
source when a few of the main power cables shorted, he said.

"We are concerned with the minimum standard for safety," he said.  
Other features such as the amount of power needed to run the resort
and backup generators on the site aren't code requirements, he said.

"It's not a safety issue. That's a business decision," he said.

Lynn said he isn't aware of any company in town that has a bulletproof
system or dual, identical power lines that run separately from Nevada
Power into a resort.

"I find it highly unlikely that any hotel has a truly redundant
(power) system," he said.

Like the Rio, the Venetian hotel-casino has also taken some extra
steps in an attempt to fend off potential power failures.

In addition to the requisite emergency power line, the Venetian owns
duplicate cabling on its property that runs into the resort, providing
some backup should one of the lines fail, Venetian's Director of
Facilities Kim Grange said. The property also has dual transformers as
well as a computerized system that allows employees to immediately
swap power loads from one line to another should problems arise, he
said.

The property could still run into problems if both power lines went
out or if Nevada Power lines failed, however, he said.

If Nevada Power lines were cut, the company's generators would only be
able to power about 60 percent of the resort, which wouldn't allow for
business as usual, Grange said.

After the Bellagio incident, Venetian engineers examined the resort's
system, which constantly undergoes preventative maintenance, he said.

"I feel that we're very protected here at the Venetian because we've
got a lot of redundant equipment," he said. "We've taken every measure
to head off any unforeseen circumstances. But there are no guarantees.  
Things can happen."

The Bellagio incident was especially unfortunate because the cable
failure occurred on a primary conduit adjacent to the resort, an
"achilles heel" for the property, Grange said.

While dual cabling at the Venetian provides some protection, the
Venetian has so far decided against building its own power plant
because Nevada Power rates are still reasonable and the cost wouldn't
be worth the return on investment, he said.

Besides giving the property immediate access to its own power source
in the event of a power outage, the Rio's cogeneration plant is also
aimed at cutting the company's power bill, Thompson said.

The plant also helps out other power users by reducing the demand for
power during peak times when power costs are higher, he said.

Some observers have questioned whether there was some flaw in the
layout of the power cables at Bellagio or whether there was some human
error involved in either the initial cable failure or the repair
process.

MGM MIRAGE spokesman Alan Feldman said the resort has until now
focused on fixing the problem and will now begin to devote more time
to investigating what caused it.

Many people have been speculating about what happened without actually
studying the evidence, which is still coming in, he said.

"Human error is one of the things we will look at," he said. "We have
focused on the evidence but not enough to make any kind of
determination about what happened. This was a complicated event. We
will likely find out that there were many things along the way."

Lynn said the county hasn't ruled out human error as a factor in the
blackout but said it's still premature to speculate about whether
something might have exacerbated the problem.

"Our primary investigation is on what initially transpired and created
the problem in the first place," he said. The county will begin its
own investigation in earnest today by examining data collected by
Bellagio computers about power supplies across the property, he said.

The reopening of the resort couldn't have happened any faster than it
did, Lynn said.

Thousands of feet of burned-out cable needed to be taken out and
replaced, a process that wasn't fully complete until Wednesday
morning, he said. The power systems then took hours to test and
retest, Lynn said.

"It takes time to put the systems back on line," he said. "We ran some
scenarios to see how they would run in an emergency system and that
took a good deal of time."

The testing process is similar to that performed at a new resort, he
said.

County building inspectors performed tests until Wednesday afternoon
and approved the property to open around 4 p.m. The property opened
just before 5 p.m. and by about 5:30 p.m. more than 1,000 guests who
had been put up at neighboring hotels had returned to check in.