[ISN] Nasdaq tests everyone's disaster recovery plans

[InfoSec News]

http://www.computerworld.com/securitytopics/security/story/0,10801,95734,00.html

By Elana Varon
SEPTEMBER 07, 2004 
CIO.com

Three years ago on Sept. 11, Kamran Rafieyan and his co-workers walked
down 83 floors of World Trade Center Tower One to safety.  
"Miraculously enough, we didn't lose any people," says Rafieyan, CIO
of Lava Trading Inc. His then-fledgling company, a service bureau that
routes equity orders for brokers, lost its data center when the tower
collapsed. "We were in the midst of building our backup site, and we
had to scramble for four weeks to get it up and running."

Today, Lava Trading counts among its clients 16 of the top 20
investment banks and helps to process 15% of the daily trading volume
on Nasdaq -- which is why on two Saturdays earlier this year, Rafieyan
joined 49 other brokers and service providers in Nasdaq-sponsored
disaster recovery tests. It was the first time that Lava Trading was
able to test its disaster plans in an everyday business setting,
rather than in a simulated environment.

In the first test, Nasdaq had customers test connectivity from their
backup sites to Nasdaq's primary site in Connecticut; in the second,
customers tested how either their primary or backup trading systems
connected to Nasdaq's backup site in Maryland. Steve Randich, Nasdaq's
executive vice president and CIO, reports nary a technical hiccup in
the entire proceedings.

The tests were the first Nasdaq offered to its entire customer base.  
(In the past, Nasdaq has accommodated individual requests for testing
whenever Nasdaq conducted its own.) With 9/11 and the August 2003
Northeast blackout behind them, it's becoming clear to many financial
services companies that their survival depends on that of their
trading partners. Regulators, meanwhile, are pushing securities
traders to have proven disaster recovery plans in place. In April, the
Securities and Exchange Commission approved a rule issued by Nasdaq's
parent company, the National Association of Securities Dealers,
requiring market participants to develop and disclose to its customers
such plans by September 2004.

Randich says his goal was "to be a host" and to allow customers to
confirm their ability to failover to their backup systems.The key
benefit, he adds, "is to promote the resilience of the market in terms
of investor protection. If there were to be a major event, we can go
to bed knowing it's not troublesome to restore operations in the
morning."

During the tests, participants worked individually with Nasdaq. Each
company was asked to test whether it could submit orders, update
quotes, submit and receive trade execution reports, and scan the
system for executed and unexecuted orders. A few companies also used
the opportunity to test their ability to send orders directly to each
other (rather than to the market as a whole), just as they would
during the course of regular business.

Disaster recovery capabilities are often an important selling point
for companies like Lava Trading that provide services to brokers. "If
you're signing a contract with a large customer for trading systems,
it involves sensitive data and mission-critical systems," notes
Rafieyan. "They want to view your fault tolerance plans [and] your
disaster recovery plans. They might ask to do regular quarterly
testing."

Through Nasdaq's tests, Rafieyan was able to confirm that his disaster
recovery plans work, but he also discovered procedures he could
improve. Combining some steps and automating others, he concluded,
would enable the company to recover more quickly from a disaster.  
Restoring service might have taken only a few minutes during a test.  
In a crisis, "It could take two to three times as long," while the
company runs through its escalation procedure to determine whether it
needs to invoke its backup plans. "That's hard to simulate," Rafieyan
says.

Collaboration in business continuity planning may be more widespread
in financial services than in other industries because financial
institutions are used to collaborating at the transaction level, says
Adrian Bowles, principal research fellow with the Robert Frances Group
Inc. consultancy. But companies in other industries could benefit from
cooperating as well.

For instance, says Bowles, most businesses rely on a few large
logistics companies to ship packages. "If [they] all go down the same
day, business stops. It would make sense from an infrastructure
standpoint to look at common failure points." Randich says Nasdaq
gives frequent tours of its data centers to visitors from Fortune 100
companies looking for business continuity advice, including a large
logistics company. "Their transactions are completely different, but
their operation and technology requirements are similar," he says.

Ultimately, maintaining business operations depends on cooperating
with more than just a company's immediate trading partners. The next
challenge, says Randich, is to be able to sustain operations if, say,
a blackout extends for 36 hours and leads into a second trading day.  
"Then telecom companies and brokers start running out of diesel fuel
to power their generators. They need transportation to help get
deliveries" and a plan for emergency personnel to help make that
happen. "The next level of improvement is a broader degree of
cooperation from the metro police and service providers."