[Infowarrior] - How Can So Much Spam Come From One Place?

[Richard Forno]

How Can So Much Spam Come From One Place?

By Brian Krebs
washingtonpost.com Staff Writer
Tuesday, November 18, 2008; 9:18 AM

http://www.washingtonpost.com/wp-dyn/content/article/2008/11/18/AR2008111801120_pf.html

At roughly 4:30 p.m. Eastern time last Tuesday, the volume of junk e- 
mail arriving at inboxes around the world suddenly plummeted by at  
least 65 percent, an unprecedented drop caused by what is believed to  
be a single, simple act.

According to security experts, one Silicon Valley based computer firm  
was playing host to computers of various organizations that controlled  
the distribution of much of the world's spam. Confronted with evidence  
tracing the spam activity back to the hosting firm, McColo Corp.,  
Internet service providers pulled the plug, severing McColo's online  
connections.

By nearly all accounts, spam volumes have remained at far diminished  
levels, though experts interviewed for this story expect spam to soon  
bounce back or even exceed previous levels. But the question remains:  
How could such a massive concentration of spam activity be hosted for  
so long from the servers at a single U.S.-based facility, in the belly  
of the security and tech community in Silicon Valley?

The answer exemplifies how complex the battle against spam has become.  
Like other Internet hosting firms, McColo -- which has not been  
charged with any crime and has been unavailable for comment -- assigns  
certain Internet addresses for its clients' computers to use. In  
effect, that's how those firms operate on the Web.

But the spam often does not come directly from those computers,  
according to security experts who have documented the activity.  
Rather, McColo appears to have been home to a number of key Internet  
servers -- computers that control networks of computers -- that were  
used by their respective owners to coordinate the actions of hundreds  
of thousands of PCs that may be compromised with malicious software  
designed to turn them into spam-spewing zombies.

According to research by several in the computer security community,  
some of the largest collections of hacked PCs, known as robot networks  
or "botnets," may have had their master control servers hosted at  
McColo. Assigned such curious monikers such as "Srizbi," "Rustock,"  
"Mega-D" and "Cutwail" by anti-virus vendors, the networks of  
compromised computers around the world are named after the malicious  
software that powers them.

The botnets typically are rented out to junk e-mail purveyors. The  
spammers then sign in remotely to those control servers and use them  
to coordinate the sending of billions of e-mails a day touting  
everything from knockoff pharmaceuticals and designer goods to  
pornography and get-rich-quick scams.

But when McColo was taken offline by its Internet providers, so, too,  
were all of the botnet control servers located there. That means  
hundreds of thousands of computers that remain infected with these bot  
programs were left like sheep without a shepherd, waiting and  
searching the Web for a new set of instructions from the criminal  
gangs that controlled them.

Joe Stewart, director of malware research for Atlanta-based  
SecureWorks, said some botnets might remain disconnected. For the  
moment, the Internet's three largest spam botnets appear to be  
stranded and unable to contact more than a small number of the their  
control servers, according to Marshal, a computer security firm in the  
United Kingdom that tracks bot activity.

Both Stewart and Marshal say the criminals responsible for maintaining  
those botnets will quickly find ways to revive them.

Not everyone has seen fewer spam messages in their inboxes after  
McColo's shutdown. Adam O'Donnell, director of emerging technologies  
at Cloudmark, an e-mail security company in San Francisco, said those  
who did not see a drop in spam from the McColo shutdown likely  
subscribe to an Internet service provider that already does an  
effective job blocking 99 percent of junk e-mail.

"People who had really good systems in place probably didn't benefit  
from this, while those who had more marginal spam filter protection  
likely saw a significant drop off in spam," O'Donnell said.

Evidence collected by anti-spam groups strongly suggests that not only  
was McColo hosting major gateways for the sending of spam, but it also  
was home to the most world's most aggressive e-mail address harvesting  
services.

In the underground spam economy, e-mail addresses are a valuable  
commodity, as they represent both the beginning and end points of any  
junk e-mail operation. Spam distribution lists typically are assembled  
using automated computer programs, or "bots," that continuously trawl  
millions of Web sites much the way that search engines do -- scouring  
them for e-mail addresses.

The addresses are then sold to spam networks, which use them as not  
only the destination for their junk e-mail, but also as the apparent  
source by "spoofing" the messages to make them appear as though they  
were sent by real, live e-mail users.

In many cases, those responsible for harvesting e-mail addresses are  
not the same people sending the spam, but rather individuals who will  
sell the lists to known spam operators.

Matthew Prince, chief executive of Unspam Technologies and founder of  
Project Honey Pot, a collaborative effort that secretly gathers  
intelligence about the world's largest spam networks, has tracked the  
spam harvesting bots hosted at McColo for more than two years.

Project Honey Pot's free technology, which is deployed at more than  
20,000 Web sites, tries to track these crawler bots by assigning a  
unique "spam trap" e-mail address to each participating site. The  
dummy addresses are designed to be difficult for humans to find but  
very easy for the bots to gather. The project's software then records  
the Internet address of any visitor and the date and time of the  
visit. Because those addresses are never used to sign up for e-mail  
lists, the software can help investigators draw connections between  
harvesters and spammers if an address generated by a spam trap or  
"honey pot" later receives junk e-mail.

Prince said statistics from Project Honey Pot suggest that crawler  
bots hosted at McColo are responsible for more than 30 million spam  
messages sent to the project's e-mail traps since June 2006.

"And our spam traps constitute a tiny fraction of the e-mail addresses  
in the world," Prince said.

The project estimates that each e-mail address harvested by bots at  
McColo could expect to receive an additional 2,000 junk e-mail  
messages a year as a result. Such activity could have major  
implications for businesses that list large numbers of employee e-mail  
addresses on their Web sites.

"Consider what this activity means for, say, a single law firm that  
publishes on its site the e-mail addresses for each of its 50  
attorneys," Prince said. "After the firm's site gets crawled by the  
bots at McColo, that means that firm can expect to receive at least  
100,000 more pieces of spam than it would have otherwise."

While there are hundreds of millions of e-mail addresses already  
registered, spammers need every address they can get their hands on  
because such a tiny percentage of people who receive the messages  
actually buy anything from them.

A study by University of California researchers released in October  
estimated that the criminals behind the Storm worm -- which powered a  
botnet once responsible for sending about 20 percent of all spam --  
made on average between $7,000 and $9,000 a day sending pharmaceutical  
spam. But the Storm worm purveyors had to send prodigious amounts of  
spam to gin up a single customer: The researchers found that while  
only about 1 in every 12 million spam e-mails turned into a sale, that  
was enough to keep the spammers in business.

Despite the level of questionable activity researchers say was coming  
out of networks hosted at McColo, it's not clear what if anything  
federal law enforcement can or should do about it, or whether anyone  
at the company has committed any crime.

A spokesman for the FBI declined to comment for this story, as did the  
U.S. Secret Service. A federal law enforcement official familiar with  
the accusations against McColo said privately that authorities have  
been investigating the hosting provider, but that building a case that  
could convince a jury of McColo's complicity in the activity has  
proven difficult.

Some in the security community, while applauding McColo's Internet  
providers for cutting the company off, said it should have happened  
sooner.

John Bambenek, incident handler with the SANS Internet Storm Center,  
which tracks hacking trends, said he doubts either provider was  
unaware of the alleged activity at McColo.

"The upstream providers may claim they didn't know, but that's about  
as convincing as a motel operator who is renting rooms by the hour and  
hearing the exploits from the hallway and being shocked when the  
police show up to bust the prostitution ring," Bambenek said.

But Benny Ng, director of infrastructure for Hurricane Electric, one  
of the Internet providers that cut off McColo's online connections,  
said that "until we were provided with the Washington Post report,  
there was no compelling overall picture." He added that many people,  
"including some professionals, think it is perfectly reasonable for an  
Internet service provider to intercept and inspect their customers  
traffic, including reading customers' email. Hurricane Electric does  
NOT condone or practice this, as this is illegal due to privacy laws."

Ng said his company monitors spam blacklists for Internet addresses  
used to send spam, but even those lists would not have flagged the  
botnet control servers hosted by McColo.

"Specifically in this case, the scope and complexity [of what was  
going on at McColo] was nearly imperceptible," said Ng. "The indirect  
nature of this network abuse, with compromised computers all over the  
world, was particularly subversive."

Global Crossing, the other major provider that pulled the plug on  
McColo's access, refused to comment.

If U.S. law enforcement was reluctant to act against McColo before the  
company's Internet providers pulled the plug, there are no signs that  
they any more willing after the incident. Sometime on Saturday,  
McColo's principals briefly reconnected the company's Web servers to a  
major Internet provider in Europe.

"The best part about this story is that they haven't physically moved  
their servers... they're still in Market Post Tower in sunny San  
Jose," at the very same Internet addresses, wrote Atif Mushtaq, a  
researcher and engineer at Fireeye.

Fireeye said the European ISP on Sunday severed its relationship with  
McColo under pressure from the security community. But that may have  
been enough time for criminals behind the Rustock botnet to reclaim  
control of between 10,000 and 15,000 of the estimated 100,000  
computers infected with the malware, Fireeye estimates.

Experts say it's not uncommon for cyber criminals to stage their  
operations out of the United States, regardless of where the criminals  
themselves may be based. After all, U.S. Internet providers offer some  
of the fastest, cheapest and most reliable Internet services on the  
planet.

"These guys like going after well-hosted infrastructure in good  
economies, because it gives them the resiliency that any business  
looks for," said Vincent Weafer, senior director of development for  
Symantec Security Response.

What's more, dependability and server uptime are important in  
cutthroat businesses for which an outage of a few hours can staunch  
the flow of spam and cost thousands of dollars.