On July 23, 2008, an article was released touting the numbers of a recent study on website security design flaws. The article only quoted some statistics from the research and did not link to it or go into detail on how the statistics were derived. I posted a quick rebuttal to the Dataloss mail list calling the entire study into question. Without the full details, the statistics really have no relevant meaning.
Shortly after the post, I was provided with a copy of the original research and was able to give a more thorough analysis of the paper. It is reproduced below, only altered for better formatting.
After being provided a link to the original paper and reading additional comments, I wanted to follow-up to my original post with more thoughts. If you want the slightly more technical review, search down to "methodology review". The paper in question is "Analyzing Websites for User-Visible Security Design Flaws" by Laura Falk, Atul Prakash and Kevin Borders. I strongly encourage more security professionals to provide peer scrutiny to security research coming from universities.
As was pointed out, the research was done in 2006 (testing in Nov/Dec) but the results are just now being published. Three people working on a study on 214 web sites should not take that long to publish. To wait so long in publishing research on a topic like this, one must question if it is responsible, or more to the point, relevant. In the world of high end custom banking applications, my experience consulting for such companies tells me that many will do periodic audits from third parties and that these sites get continuous improvements and changes every week. One of the web sites I use for personal banking has changed dramatically in the last 12 months, making huge changes to the functionality and presumably architecture, security and design. The results of a 2006 audit of that site are probably most irrelevant.
As with most research papers, the lack of publish date in the header is annoying. The abstract does not mention the 2006 to 2008 time gap between research and publication either. This time difference is seen almost immediately in the citation of Schechter et al, regarding people "disregarding SSL indicators". The current releases of several browsers, most notably IE7 and Firefox 3 make pretty big shifts in how the browser handles and warns about SSL indicators. Each browser is considerably more paranoid and will throw a warning over more discrepencies that each would have ignored in previous versions.
On page 1, Prakash et al list the criteria for the categories of "design flaws" they examined. As expected and mentioned in my previous post, the design flaws they examined are not necessarily a vulnerability, and often times do not put customer data at risk or they require additional requisites to be exploitable. To look at one of their design flaws as an example, consider the following:
Presenting secure login options on insecure pages: Some sites present login forms that forward to a secure page but do not come from a secure page. This is problematic because an attacker could modify the insecure page to submit login credentials to an insecure destination.
This summary of a design flaw is problematic in that it makes several assumptions and/or does not fully qualify the attack vector. First, to "modify" an insecure page being served from the bank to the user's client (browser), the attacker would have to compromise the server (making this attack moot) or conduct a Man-in-The-Middle (MiTM) attack. I assume the latter is meant since the implication is that an attacker could not effectively MiTM attack a page wrapped in encryption (SSL).
It is interesting that the lack of SSL encryption is chosen as a design flaw with the notion that manipulation of an insecure page is the preferred attack vector. Such an attack is considerably more difficult to conduct compared to other threats (e.g. SQL Injection, Privilege Escalation) and would essentially target a single customers. Many large applications serving hundreds of thousands of users makes this trade-off of mixed security pages for performance reasons, as the overhead of encrypting all traffic can be costly.
Later in the paper when the team attempts to better define this design weakness, they say:
Consider the case where the customer service contact information for resetting passwords is provided on an insecure page. To compromise the system, an attacker only needs to spoof or modify the page, replacing the customer service phone numbers with bogus numbers.
Web pages can be spoofed regardless of the transport, so the presence of SSL encryption means little to nothing. If his team is implying an attacker "only need [..] modify the page", that would require compromising the server or performing a MiTM attack. Again, this is not a trivial attack by any means and in the latter, would affect one customer.
While this is only one of five design flaws Prakash's team looked for, consider the third example which is the exact same design weakness:
Contact information/security advice on insecure pages: Some sites host their security recommendations, contact information, and various other sensitive information about their site and company on insecure pages. This is dangerous because an attacker could forge the insecure page and present different recommendations and contact information.
This is the exact same issue as #2 in the list but just makes the specification of the content on the page. Factor in that issue #1 will be more prevalent in large organizations but a non-issue in smaller ones and the criteria of five design weaknesses gets cut down from five to four, with one that is likely not to be seen on some of the sites tested at all.
The paper quickly summarizes their findings before going in detail, before concluding "Overall, only 24% of the sites were completely free of these design flaws, indicating that some of the flaws we identified are not widely understood, even among institutions where security is critical." This assumption and conclusion is dangerous and irresponsible. The implication that the presence of one or more of these flaws is indicative of the site not understanding the threat is presumptious. With the example given above about the high overhead of encrypting all content, some of the "design flaws" may be business decisions and acceptable risk.
Prakash et al begin to demonstrate their lack of understanding of client-server relationships and the transport mechanism for different protocols. The following paragraph from page 2 immediately calls his team's technical competence:
One of the most interesting design flaws we discovered is the presentation of FAQs and contact information on insecure pages. In the past, FAQs and contact information were usually sent through the mail to the customer. It is not generally recognized that this information should be protected. However, when this information is presented online, the user becomes vulnerable to socialengineering and offline attacks as a result of the information being displayed on an insecure page.
Prakash's contention that unencrypted content delivered from a web server to a browser is somehow different than when unencrypted content is delivered from a mail server to a mail reader is silly. If an attacker has the ability to MiTM attack a person, it isn't going to be limited to HTTP. Sending that contact information via mail will result in a user deleting it or maybe storing it in a folder. The first time the person needs to contact the bank, they would check the web page for the contact information. If said information is not available, it now further burdens the bank as they may call a generic number and get transferred around several times. This adds to customer frustration and causes bank employees to spend extra time dealing with a customer that could have called the correct number to begin with.
Prakash's team goes on to make more assumptions or not fully understand the importance of how web clients behave. Without getting into a full discussion on the philosophy of e-commerce sites adding mechanisms to invalidate client-side vulnerabilities, the general notion that it should be done if feasible seems reasonable. In this context, feasible means that it doesn't overly burden the bank web site, does not impact performance and is generally transparent to the end user. One example of this is a Cross-frame spoofing issue that made it trivial for an attacker to use a phishing attack to MiTM attack MSIE 6 users. Web sites can add a small bit of javascript to help ensure that browsers load their pages in a new frame and essentially mitigates this risk. This is a good example of how many banks were helping protect customers, even though the vulnerability was in the customer's software, not the bank web site. Prakash's team claims:
Our work is similar in that some of the flaws that we consider impair a user's ability to make correct security decisions. However, our work differs in that the cause is not poor or confusing client-side interfaces. Instead, the flaws originate in poor design or policy choices at the server that prevent or make it difficult for users to make correct choices from the perspective of securing their transactions.
While a mismatched SSL certificate used to be virtually ignored in some cases, new versions of popular browsers now behave differently in how they alert users, giving them the ability to more easily make correct choices. Claiming that this research is not impacted by "poor or confusing client-side interfaces" is misleading. While the older browsers were not necessarily confusing, they handled some situations regarding establishing trust poorly.
The next area of technology Prakash's team doesn't seem to fully understand is vulnerability scanners. In the paper his team says:
Network scanners, such as Nessus [11], and application-level website scanners, such as AppScan [17], can be used to analyze for many configuration and implementation bugs, such as use of unpatched services and vulnerability to cross-side scripting or SQLinjection attacks. As far as we are aware, the design flaws that we examine are currently not identified by these scanners.
Both Nessus and AppScan will identify several vulnerabilities that directly relate to the design flaws outlined. Both will give warning over invalid or expired SSL certificates, AppScan will warn about mixed-mode security pages and neither will perform tests for some of the design flaws listed (#3, #4, #5) because no scanner in the world can do it.
Methodology Review:
On page 8 (of 10), the team gives very brief descriptions of their testing methodology. The lack of description or their testing methodology undermines significant portions of the research. For "Break in the Chain of Trust", the paper says "Under no circumstance should an insecure page make a transition to a securitysensitive website hosted on another domain, regardless of whether the destination site uses SSL." This is an arbitrary 'rule' that is not widely accepted by anyone including the banking industry. Many web pages are designed to act as portals that link to additional features. The 'rule' as quotes from the paper would force large bank organizations to consolidate all web resources on a single domain. While that may be nice, it simply isn't feasible to many businesses, especially ones with a large organization that includes multiple companies. Linking from http://bigbank.com/ to https://regionalbank.com/ is perfectly acceptable and should use proper SSL certificates and technology controls to help ensure the user ends up on the correct page, loaded directly by the browser.
The second design weakness studied was "Presenting Secure Login Options on Insecure Pages". The paper explains their methodology as ".. searched each web page for the string "login". If the string was found, we searched the same page for the strings "username" or "user id" or "password". If the string .login. and .username. or .user id. or .password. were found on the same page, we then verified whether the page was displayed using the HTTP protocol. If this was the case, we assumed this site contained the design flaw." The key word here being 'assumed'. There are scenarios where the above methodology could easily generate a false positive. Even back in 2006, there were trivial ways to more easily determine the use of HTTPS with certainty.
The third design weakness studied was "Contact Information/Security Advice on Insecure Pages" and is perhaps the most technically lacking testing method one could perform:
We searched each web page for the string "contact", "information", or "FAQ". If those strings where found, we checked whether the page was protected with SSL. If not, then we considered it to contain the design flaw.
The mere presence of these words on a site do not mean they are in the context of listing bank contact information. While 'contact' will frequently link to a 'contact us' page, looking for 'information' or 'FAQ' is absurd.
In the fourth weakness, "Inadequate Policies for User IDs and Passwords", the team openly admits that their methodology may produce "optimistic" results and that they had no way to verify their results "without generating an account on the website". Heaven forbid they find a couple hundred students at the university to participate by logging into their personal banks and checking this in more detail. That extra effort would have made this portion the only positive and accurate test. From the paper:
Our count could be optimistic; some sites may require strong passwords without stating an explicit policy. We had no obvious means of verifying this without generating an account on the website. Our count could also be conservative for sites that have poor policies resulting in weak passwords. Thus, our results for this design flaw should only be taken as a rough estimate of the extent of this particular problem.
As before, the fifth design weakness was extrapolated using a glorified 'grep' of the web page, analyzing proximity of a few keywords and then verifying the hits above an 85% threshold. And as before, this testing methodology makes huge assumptions about the wording on the page, does not positively account for HTML formatting that would impact the 'distance' between words (especially in pages with frames) and does not begin to test the functionality (see page 9, section 4.5).
Finally, the paper attempts to interpret the results of this poorly conceived and improperly tested study. Table 1 on page 10 says that 30% of the sites tested were affected by "Break in the chain of trust" but gets contradicted (clarified) in the first paragraph of the results:
With automated tools, such as the one used in our study, false positives are possible. To the extent feasible, we manually examined the results to eliminate false positives from the reported data. Our break-in-chain-of-trust data had a significant number false positives. Our automated tool reported about 30% of the websites to potentially use third-party sites in an unsafe way, but only 17% were found to do so without giving some sort of notification to the user about that transition.
Having such an admittedly large margin of error on the results based on their own methodology should be an eye-opener in regards to the integrity and accuracy of the results. Despite this revelation, the next paragraph immediately cites the table with the 30% number and begins to make conclusions based on the bogus numbers. They go on to further explain their primary tool for gathering information (the wget tool) may not have retrieved all of the information needed to properly assess the site.
Despite the weak methodology, 44% error rate on at least one test and admitted errors, Prakash et al go on to say "We found that 76% of sites have at least one design flaw." Such statements are certainly not factual or even statistically correct based on the research presented.
Prakash followed up with a brief reply that was dismissive of my write-up, and only called for more research of bank sites. What he failed to understand is that more research of this nature, with huge gaps and a lack of understanding of the subject matter, will only produce more statistics and articles that have no relevance or meaning in our industry. Worse, they serve to provide inaccurate information that could only be used as a basis for poor decisions for people with an interest in information security.
Copyright 2008-2010 by Brian Martin. Permission is granted to quote, reprint or redistribute provided the text is not altered, and appropriate credit is given.