[Infowarrior] - NSA rejected system that sifted phone data legally

[Richard Forno]

NSA rejected system that sifted phone data legally
Dropping of privacy safeguards after 9/11, turf battles blamed
By Siobhan Gorman
Sun reporter
Originally published May 18, 2006
http://www.baltimoresun.com/news/custom/attack/bal-te.nsa18may18,0,4406058.s
tory?coll=bal-home-headlines

WASHINGTON // The National Security Agency developed a pilot program in the
late 1990s that would have enabled it to gather and analyze huge amounts of
communications data without running afoul of privacy laws. But after the
Sept. 11 attacks, it shelved the project - not because it failed to work but
because of bureaucratic infighting and a sudden White House expansion of the
agency's surveillance powers, according to several intelligence officials.

The agency opted instead to adopt only one component of the program, which
produced a far less capable and rigorous program. It remains the backbone of
the NSA's warrantless surveillance efforts, tracking domestic and overseas
communications from a vast databank of information, and monitoring selected
calls.

Four intelligence officials knowledgeable about the program agreed to
discuss it with The Sun only if granted anonymity because of the sensitive
nature of the subject.

The program the NSA rejected, called ThinThread, was developed to handle
greater volumes of information, partly in expectation of threats surrounding
the millennium celebrations. Sources say it bundled four cutting-edge
surveillance tools. ThinThread would have:

€ Used more-sophisticated methods of sorting through massive phone and
e-mail data to identify suspect communications.

€ Identified U.S. phone numbers and other communications data and encrypted
them to ensure caller privacy.

€ Employed an automated auditing system to monitor how analysts handled the
information, in order to prevent misuse and improve efficiency.

€ Analyzed the data to identify relationships between callers and chronicle
their contacts. Only when evidence of a potential threat had been developed
would analysts be able to request decryption of the records.

An agency spokesman declined to discuss NSA operations.

"Given the nature of the work we do, it would be irresponsible to discuss
actual or alleged operational issues as it would give those wishing to do
harm to the U.S. insight and potentially place Americans in danger," said
NSA spokesman Don Weber in a statement to The Sun. "However, it is important
to note that NSA takes its legal responsibilities very seriously and
operates within the law."

In what intelligence experts describe as rigorous testing of ThinThread in
1998, the project succeeded at each task with high marks. For example, its
ability to sort through huge amounts of data to find threat-related
communications far surpassed the existing system, sources said. It also was
able to rapidly separate and encrypt U.S.-related communications to ensure
privacy.

But the NSA, then headed by Air Force Gen. Michael V. Hayden, rejected both
of those tools, as well as the feature that monitored potential abuse of the
records. Only the data analysis facet of the program survived and became the
basis for the warrantless surveillance program.

The decision, which one official attributed to "turf protection and empire
building," has undermined the agency's ability to zero in on potential
threats, sources say. In the aftermath of revelations about the agency's
wide gathering of U.S. phone records, they add, ThinThread could have
provided a simple solution to privacy concerns.

A better system
A number of independent studies, including a classified 2004 report from the
Pentagon's inspector-general, in addition to the successful pilot tests,
found that the program provided "superior processing, filtering and
protection of U.S. citizens, and discovery of important and previously
unknown targets," said an intelligence official familiar with the program
who described the reports to The Sun. The Pentagon report concluded that
ThinThread's ability to sort through data in 2001 was far superior to that
of another NSA system in place in 2004, and that the program should be
launched and enhanced.

Hayden, the president's nominee to lead the CIA, is to appear today before
the Senate Select Committee on Intelligence and is expected to face tough
questioning about the warrantless surveillance program, the collection of
domestic phone records and other NSA programs.

While the furor over warrantless surveillance, particularly the collection
of domestic phone records, has raised questions about the legality of the
program, there has been little or no discussion about how it might be
altered to eliminate such concerns.

ThinThread was designed to address two key challenges: The NSA had more
information than it could digest, and, increasingly, its targets were in
contact with people in the United States whose calls the agency was
prohibited from monitoring.

With the explosion of digital communications, especially phone calls over
the Internet and the use of devices such as BlackBerries, the NSA was
struggling to sort key nuggets of information from the huge volume of data
it took in.

By 1999, as some NSA officials grew increasingly concerned about
millennium-related security, ThinThread seemed in position to become an
important tool with which the NSA could prevent terrorist attacks. But it
was never launched. Neither was it put into effect after the attacks in
2001. Despite its success in tests, ThinThread's information-sorting system
was viewed by some in the agency as a competitor to Trailblazer, a $1.2
billion program that was being developed with similar goals. The NSA was
committed to Trailblazer, which later ran into trouble and has been
essentially abandoned.

Both programs aimed to better sort through the sea of data to find key tips
to the next terrorist attack, but Trailblazer had more political support
internally because it was initiated by Hayden when he first arrived at the
NSA, sources said.

NSA managers did not want to adopt the data-sifting component of ThinThread
out of fear that the Trailblazer program would be outperformed and
"humiliated," an intelligence official said.

Without ThinThread's data-sifting assets, the warrantless surveillance
program was left with a sub-par tool for sniffing out information, and that
has diminished the quality of its analysis, according to intelligence
officials.

Sources say the NSA's existing system for data-sorting has produced a
database clogged with corrupted and useless information.

The mass collection of relatively unsorted data, combined with system flaws
that sources say erroneously flag people as suspect, has produced numerous
false leads, draining analyst resources, according to two intelligence
officials. FBI agents have complained in published reports in The New York
Times that NSA leads have resulted in numerous dead ends.

Privacy safeguards
The privacy protections offered by ThinThread were also abandoned in the
post-Sept. 11 push by the president for a faster response to terrorism.

Once President Bush gave the go-ahead for the NSA to secretly gather and
analyze domestic phone records - an authorization that carried no
stipulations about identity protection - agency officials regarded the
encryption as an unnecessary step and rejected it, according to two
intelligence officials knowledgeable about ThinThread and the warrantless
surveillance programs.

"They basically just disabled the [privacy] safeguards," said one
intelligence official.

A former top intelligence official said that without a privacy requirement,
"there was no reason to go back to something that was perhaps more difficult
to implement."

However two officials familiar with the program said the encryption feature
would have been simple to implement. One said the time required would have
involved minutes, not hours.

Encryption would have required analysts to be more disciplined in their
investigations, however, by forcing them to gather what a court would
consider sufficient information to indicate possible terrorist activity
before decryption could be authorized.

While it is unclear why the agency dropped the component that monitored for
abuse of records, one intelligence official noted that the feature was not
popular with analysts. It not only tracked the use of the database, but
hunted for the most effective analysis techniques, and some analysts thought
it would be used to judge their performance.

Within the NSA, the primary advocate for the ThinThread program was Richard
Taylor, who headed the agency's operations division. Taylor who has retired
from the NSA, did not return calls seeking comment.

Officials say that after the successful tests of ThinThread in 1998, Taylor
argued that the NSA should implement the full program. He later told the
9/11 Commission that ThinThread could have identified the hijackers had it
been in place before the attacks, according to an intelligence expert close
to the commission.

But at the time, NSA lawyers viewed the program as too aggressive. At that
point, the NSA's authority was limited strictly to overseas communications,
with the FBI responsible for analyzing domestic calls. The lawyers feared
that expanding NSA data collection to include communications in the United
States could violate civil liberties, even with the encryption function.

Taylor had an intense meeting with Hayden and NSA lawyers. "It was a very
emotional debate," recalled a former intelligence official. "Eventually it
was rejected by [NSA] lawyers."

After the 2001 attacks, the NSA lawyers who had blocked the program reversed
their position and approved the use of the program without the enhanced
technology to sift out terrorist communications and without the encryption
protections.

The NSA's new legal analysis was based on the commander in chief's powers
during war, said former officials familiar with the program. The Bush
administration's defense has rested largely on that argument since the
warrantless surveillance program became public in December.

The strength of ThinThread's approach is that by encrypting information on
Americans, it is legal regardless of whether the country is at war,
according to one intelligence official.

Officials familiar with Thin Thread say some within NSA were stunned by the
legal flip-flop. ThinThread "was designed very carefully from a legal point
of view, so that even in non-wartime, you could have done it legitimately,"
the official said.

In a speech in January, Hayden said the warrantless surveillance program was
not only limited to al-Qaida communications, but carefully implemented with
an eye toward preserving the Constitution and rights of Americans.

"As the director, I was the one responsible to ensure that this program was
limited in its scope and disciplined in its application," he said.

siobhan.gorman at baltsun.com