[Dataloss] seriously flawed U Washington breach study

[Adam Shostack]

On Wed, Mar 14, 2007 at 05:35:33PM -0500, Bill Yurcik wrote:
| 
| On Wed, 14 Mar 2007, Adam Shostack wrote:
| > On the other hand, they could definetly have been more clear about the
| > difference between 0 breaches and 0 reported breaches.
| 
| the authors did not identify (maybe because they did not recognize) how 
| incredibly bad their data is (years of data that are not even close),
| they then went on to make bold claims! trash-in trash-out

On "page 22 of 31," starting from line 37:

> Several factors might explain the pattern of increasing incidents
> and volume of compromised data over time. First, there is the
> possibility that the results are skewed due to the relative growth
> of new, fresh news stories devoted to this issue, and the loss of
> older stories that disappeared from news archives as time
> passed. Perhaps there have always been hundreds of incidents every
> year, but only in recent years has the severity of the problem been
> reported in the news. If this were the case, we would expect to see
> a gradually decaying pattern with greater number of reported cases
> in 2006 than in 2005, 2004, and so on. However, the dramatic
> difference in reported incidents between later years and early years
> suggests that this effect does not adequately explain ...

So I'm confused by your claim that they don't recognize the issue.  

Adam

| > On Wed, Mar 14, 2007 at 03:32:40PM -0500, Bill Yurcik wrote:
| > |
| > | "Hackers Get a Bum Rap for Corporate America's Digital Delinquency"
| > | University of Washington News and Information (03/12/07)
| > | http://uwnews.washington.edu/ni/article.asp?articleID=31264
| > |
| > | I saw this press announcement of a study (also included in summary at end
| > | of this Email) getting publicity and it looks seriously flawed. The
| > | academics searched news articles about computer breaches going back to
| > | 1980 and then make claims.
| > |
| > | (1) the authors, who are not techies (communications and geography
| > | academics), should realize that there are significant disincentives for
| > | any organization to have breaches of any type publicly reported - this
| > | makes any aggregate news data about breaches they assembled extremely
| > | suspect.
| > |
| > | for instance, the authors claim there were *zero* breaches each year for
| > | the years 1988-91, 1993-94; less than 10 breaches each year from
| > | 1995-1999; and less than 25 breaches each year from 2000-2004.
| > | this does not pass the smell test!!!
| > |
| > | (2) I would also argue only since state breach disclosure laws have
| > | started to provide accurate data on "privacy breaches" can one begin to
| > | make claims - there is not valid data before state disclosure laws kicked
| > | in.  Even state breach disclosure data is relatively new to being
| > | analyzed and not perfect since there is still non-reporting and
| > | disclosures are not publicly recorded although the press does pick up a
| > | significant portion of the disclosures between organizations and the
| > | parties affected. Also there are skewing effects due to state
| > | breach disclosure laws not being uniform and having different technical
| > | requirements such as who must report, what they must report, etc.
| > |
| > | (3) The study in question mixes news events with
| > | recent reports to comply with state disclosure laws so this changes any
| > | statistical analysis (multiple sources from different distributions)
| > |
| > | I am very disappointed to see this poor scholarship/analysis
| > | especially that it is getting press (primarily due to the University of
| > | Washington's public relations).  Of course consider the source where the
| > | study will evemtually be published is not at the forefront in
| > | this area, "Journal of Computer-Mediated Communication", however, due
| > | dilligence should have sent the editors of JCMC to seek out some of us
| > | from this dataloss list for peer-review.
| > |
| > | any feedback in agreement or disagreement?
| > |
| > | Cheers! - Bill Yurcik
| > |
| > | ---
| > |
| > | "Hackers Get a Bum Rap for Corporate America's Digital Delinquency"
| > | University of Washington News and Information (03/12/07)
| > | http://uwnews.washington.edu/ni/article.asp?articleID=31264
| > |
| > | University of Washington communications professor Phil Howard conducted a
| > | review of data-breach incidents reported in major U.S. news outlets between
| > | 1980 and 2006 and found that organizational flaws in businesses, not
| > | hackers, should receive the most blame.  "The surprising part is how much
| > | of those violations are organizationally prompted--they're not about lone
| > | wolf hackers doing their thing with malicious intent," Howard says.  His
| > | study revealed that malicious intrusions represent only 31 percent of 550
| > | confirmed incidents, while mismanagement, such as missing or stolen
| > | hardware, insider abuse or theft, administrative errors, or accidental
| > | exposure of data online was responsible for 60 percent of the incidents
| > | reported.  State laws that require companies to report breaches enabled the
| > | study to be done with greater accuracy.  "We've actually been able to get a
| > | much better snapshot of the spectrum of privacy violations," says Howard.
| > | The study also found that while universities make up less than 1 percent of
| > | the total records lost, they make up 30 percent of the reported incidents.
| > | Corporate America claims that market forces should be allowed to solve the
| > | problem of data breaches and reporting them, but Howard believes that this
| > | strategy is not sufficient, especially since identity theft is the nation's
| > | fastest growing crime.  He also believes that states seem more capable of
| > | passing laws on the matter than the federal government.
| > |
| > | ---
| > | _______________________________________________
| > | Dataloss Mailing List (dataloss at attrition.org)
| > | http://attrition.org/dataloss
| > | Tracking more than 149 million compromised records in 598 incidents over 7 years.
| >
| _______________________________________________
| Dataloss Mailing List (dataloss at attrition.org)
| http://attrition.org/dataloss
| Tracking more than 149 million compromised records in 598 incidents over 7 years.