[Infowarrior] - How a stray mouse click choked the NYSE & cost a bank $1

[Richard Forno]

How a stray mouse click choked the NYSE & cost a bank $150K
By Jon Stokes | Last updated January 27, 2010 8:51 PM

http://arstechnica.com/business/news/2010/01/how-a-stray-mouse-click-choked-the-nyse-cost-a-bank-150k.ars
fat fin·ger |fat 'fi NG gər |, verb, to enter a trade order  
incorrectly into an electronic stock trading system: He fat-fingered  
that sell order by accidentally adding two zeroes.

The term comes from the idea of a clumsy, "fat-fingered" typist, who  
presses extra keys without being aware of it.

As the practice of high-frequency trading continues to become more  
widespread, concerns are growing that erroneous trades carried out by  
"algos gone wild"—a sort of digitally amplified version of the "fat  
finger" phenomenon—could cause a market crash at Internet speed, a  
meltdown that no one could stop. Two recent market glitches could  
provide a preview of what's to come.

Double-click of death: the Credit Suisse fiasco
Like Goldman, Sachs and other large banks, Credit Suisse has a  
proprietary trading desk—i.e., a division of the bank that trades  
stocks and derivatives in order to make money for the bank itself  
(instead of for the bank's customers). And, like other banks in its  
class, much of this proprietary trading is now done entirely by  
computers that execute trades by the millisecond based on tiny,  
fleeting fluctuations in a stock's price.

On November 14, 2007 at 3:20pm one of Credit Suisse's trading  
algorithms suddenly went haywire, and, in a few moments, sent hundreds  
of thousands of bogus requests to the exchange. This sudden surge of  
requests, which were cancellations for a large batch of orders that  
the machine had never actually sent out, acted like a denial-of- 
service attack on some parts of the New York Stock Exchange. The  
messages clogged the tubes and caused parts of the exchange to freeze  
up, affecting trading in 975 stocks.

After an extensive investigation, the NYSE assessed a $150,000 fine  
for Credit Suisse's "failing to adequately supervise the development,  
deployment and operation of a proprietary algorithm, including a  
failure to implement procedures to monitor certain modifications made  
to the algorithm."

The exchange's filing, released a little over a week ago, has the  
details of precisely what drove the algorithm haywire—it was a trader  
who accidentally double-clicked an icon in a trading program's  
interface, when he should've single-clicked. No, I am not making that  
up.

And you thought butterfly ballots were bad UI design
The algorithm that choked is part of a program called SmartWB, and it  
was designed to spit out a constant stream of trading orders all day  
long, using a set of fixed parameters. These orders would be  
transmitted to the NYSE, where they would go into a queue and wait to  
be executed.

A few days prior to the incident, a programmer took it upon himself to  
unilaterally improve SmartWB by adding a new user input feature, which  
would let a trader change a certain parameter on-the-fly. When a user  
changed the parameter, all the new orders that came out of SmartWB  
would reflect the change, and all of the existing orders in the queue  
waiting to be executed would instead be cancelled and replaced with  
otherwise identical orders that contained the tweaked parameter.

The interface for inputting this new parameter was really simple—and  
really, really stupid. There was a box for the trader to enter a  
number, and a pair of arrows next to the box: "up" and "down."  
Pressing the "up" arrow would revise the parameter upwards by the  
amount in the box, and pressing the "down" arrow would revise it  
downwards. After you clicked an arrow button, the new parameter would  
instantly take effect for all new orders, and all existing, queued  
orders would be cancelled and replaced with orders that reflected the  
change—no "Are you sure?" dialog box or any other form of sanity  
check, just instant execution.

The lack of feedback and "forgiveness" in the interface element would  
have been bad enough, but there was another problem, and one that  
would have been uncovered during testing... if there had been any  
testing, which there had not.

On November 14, a few seconds after 3:20, a trader put a number in the  
box and then double-clicked the "up" arrow. This double-click was  
interpreted by SmartWB as two separate clicks, so the system dutifully  
sent out a second batch of cancel/replace orders in addition to the  
batch that was intended by the trader. This sudden flood of cancel/ 
replace orders, half of which were requesting cancellation of orders  
that had never been sent, overwhelmed the system and backed up five of  
the posts on the NYSE trading floor.

The incident wasn't a major catastrophe by any measure—it cost money,  
and overloaded a few posts near the close of the trading day, but  
otherwise it had no lasting effects. And the errors involved—a stray  
click, a bad UI decision, failures of testing and oversight—were all  
human errors, albeit amplified many times over by the speed and power  
of the network. You might even say that trader's the stray click was  
like the proverbial flutter of a butterfly's wing in China, except it  
didn't quite cause a hurricane in the Atlantic—just a nasty downpour.

Since 2007, our markets have moved drastically further in the  
direction of complete automation. But it's not clear that Wall  
Street's programmers have made correspondingly large leaps in testing,  
UI design, and version control. Indeed, on message boards and blogs,  
day traders who follow the market tick-by-tick swap stories of huge  
swings in a stock price, where a stock plunges or spikes but then  
corrects in a few minutes, after the exchange realizes there was an  
error and cancels the trades.

These glitches are typically attributed to high-frequency trading  
algorithms gone temporarily insane—"algos gone wild" is the preferred  
phrase. It is suspected that Rambus's stock was the victim of of an  
algorithm-driven selloff early this year.

The Rambus incident
On the afternoon of January 4, 2010, Rambus's stock lost 35 percent of  
its value in a matter of seconds. NASDAQ ended up canceling all trades  
at or below $20.73, and blamed the error on some unspecified trader's  
"fat finger."

Anecdotally, bizarre market action like this is becoming a fact of  
life for day traders, who now have to take extra measures to guard  
against being hurt by these moves. For instance, many day traders use  
automated "trailing stops" if they're going to be away from a terminal  
for a few hours—these stops dictate that if a stock drops by a  
certain amount, then the trader's platforms should put in an automated  
sell order to  dump the security before it goes down even further.

If a trader was using an automated trailing stop on Rambus on January  
4th, his system would've dumped the stock automatically, incurring a  
trading fee and possibly even losing money, only to see the stock jump  
back up again.

Of course, the fact that such computer-driven volatility hurts day  
traders matters little to long-term investors. But the fear is that  
these glitches are fleeting indications that the system as a whole is  
vulnerable and unstable, and that the right combination of  
circumstances could cause what happend to RMBS to happen on a wider  
scale. This is especially true as even more of the trading activity,  
even among individual traders, shifts to automated platforms.

Epilogue: From mainframe to PC
One trend of the past year that has been covered at the Financial  
Times and other outlets is the democratization of computer-automated  
trading. Small two- and three-person trading shops, manned by a laid- 
off quant or two and a programmer, are cropping up all over the  
country and combining the latest gaming-oriented GPU and CPU hardware  
with standard electronic trading accounts to do algorithmic trading on  
a small scale.

Because these groups are using commodity gaming hardware to generate  
"buy" and "sell" orders that are then transmitted to the market using  
standard retail trading software, there's no way to tell them apart  
from normal (human) day traders. Thus, there's absolutely no way to  
gauge how large of a phenomenon this actually is.

The Obama administration's efforts to rein in high-frequency trading  
by eliminating flash orders and banning proprietary trading (much of  
which is HFT-based) from large banks will probably have the effect of  
leveling the playing field a bit for these smaller algo shops. As  
Matthew Goldstein at points out in his Reuters article on the topic,  
the prop desks may disappear, but the software and expertise will not.  
Instead of being concentrated at a few large banks, algo trading will  
just spread, as the talent behind it either jumps to new funds or goes  
solo.

And if, thanks to further government intervention, the big boys  
ultimately lose some of the latency advantage that their deep pockets  
buy them, then the future of the market could well belong to hundreds  
of thousands of small trading operations who, instead of playing  
Crysis on their top-of-the-line Alienware systems, use them to print  
money instead.