This is a rebuttal to Why Antivirus Companies Like Mine Failed to Catch Flame and Stuxnet (June 1, 2012) by Mikko Hypponen. There are several updates to this article at the end, based on replies from a variety of people including Mikko.
The antivirus market is worth over 4 billion dollars, with a 'B'. Oh, one other thing; that was six years ago. I lead off with this point because a lot of arguments about the antivirus world can be tracked back to this. [Note: I tend to use 'malware' more than 'virus', but interchange depending on circumstances.]
I like Mikko Hypponen and I think that overall, he typically offers a level-headed perspective on security, especially when it comes to antivirus topics. Mikko is the Chief Research Officer of F-Secure, a company that is founded on antivirus technology. When it comes to malware, Mikko should be an expert and he should give us a rational perspective. This time however, he has failed spectacularly. I offer these counter points to his article in the hopes of shedding light on his bullshit, and the bullshit that the antivirus industry has been feeding us for a long, long time.
A couple of days ago, I received an e-mail from Iran. It was sent by an analyst from the Iranian Computer Emergency Response Team, and it was informing me about a piece of malware their team had found infecting a variety of Iranian computers. This turned out to be Flame: the malware that has now been front-page news worldwide.
When we went digging through our archive for related samples of malware, we were surprised to find that we already had samples of Flame, dating back to 2010 and 2011, that we were unaware we possessed.
In the second paragraph of this bizarre apology letter, Mikko Hypponen clearly states that the antivirus company he works for found or detected Flame, as far back as 2010. In this very same article, he goes on to say that F-Secure and the antivirus industry failed to detect Flame three times, while saying that they simply can't detect malware like Flame. Huh? How did he miss this glaring contradiction? How did Wired editors miss this?
The fact is, your research team or software flagged Flame as malware two years ago. It sat in your archive, ignored all this time. Any claims that you "couldn't" have or "shouldn't" have detected it are wrong and disingenuous.
Researchers at other antivirus firms have found evidence that they received samples of the malware even earlier than this, indicating that the malware was older than 2010.
What this means is that all of us had missed detecting this malware for two years, or more.
Rage. Epic levels of rage at how bad this spin doctoring attempt is, and how fucking stupid you sound. YOU DETECTED THE GOD DAMN MALWARE. IT WAS SITTING IN YOUR REPOSITORY.
It wasn't the first time this has happened, either. Stuxnet went undetected for more than a year after it was unleashed in the wild, and was only discovered after an antivirus firm in Belarus was called in to look at machines in Iran that were having problems.
For those of us in the world of security, hearing an antivirus company say "we missed detecting malware" isn't funny, because the joke is so old and so very tragically true. The entire business is based on a reactionary model, where bad guys write malware, and AV companies write signatures sometime after it. For those infected by the malware, it's too little too late. It's like a couple of really inept bodyguards, who stand next to you while you're getting beaten up and say, "I will remember that guy's face next time and ask management to not let him in." Welcome to the world of antivirus.
When researchers dug back through their archives for anything similar to Stuxnet, they found that a zero-day exploit that was used in Stuxnet had been used before with another piece of malware, but had never been noticed at the time.
This statement is a red herring Mikko. Stuxnet used five vulnerabilities, only one of which was 0-day. One local privilege escalation used in Stuxnet was unknown at the time, the rest were documented vulnerabilities. If your excuse for the AV industry is that Stuxnet wasn't detected because it used a 0-day, your argument falls flat in that you should have detected the other three code execution / privilege escalation vulnerabilities (the fifth being default credentials).
Stuxnet, Duqu and Flame are not normal, everyday malware, of course. All three of them were most likely developed by a Western intelligence agency as part of covert operations that weren't meant to be discovered.
What, because all of the other malware was meant to be discovered? A big DERP on that one Mikko.
The fact that the malware evaded detection proves how well the attackers did their job.
Again, it didn't evade detection, according to Mikko Hypponen, Chief Blah Blah at an Antivirus Company. He said so in the second paragraph of an article I read. That said, the fact is antivirus companies miss hundreds of pedestrian malware samples every day. Is this because the authors did so well, or that your business model and detection capabilities are flawed? One could easily argue that they are intentionally so (reference that bit about 4 billion dollars).
And instead of trying to protect their code with custom packers and obfuscation engines - which might have drawn suspicion to them - they hid in plain sight. In the case of Flame, the attackers used SQLite, SSH, SSL and LUA libraries that made the code look more like a business database system than a piece of malware.
Where to begin. First, custom packers and obfuscation engines have worked very well against antivirus software for a long time. I don't think that would have drawn any more suspicion. Second, Flame is 20 megs, around 20x more code than Stuxnet. In the world of antivirus, where you are usually scanning very small bits of obfuscated code, this should seem like a godsend. If it isn't using obfuscation, then what is the excuse for missing it? Are you really telling me that your industry is just now realizing the "hide in plain sight" method, in 2012?
Someone might argue that it's good we failed to find these pieces of code. Most of the infections occurred in politically turbulent areas of the world, in countries like Iran, Syria and Sudan. It's not known exactly what Flame was used for, but it's possible that if we had detected and blocked it earlier, we might have indirectly helped oppressive regimes in these countries thwart the efforts of foreign intelligence agencies to monitor them.
Another compelling reason why people shouldn't be mad at antivirus, because in failing to detect Stuxnet and Flame, you may have helped Western intelligence agencies carry out vital operations against foreign oppressive regimes. Bringing up politics in this argument is a clever and interesting angle!
But that's not the point. We want to detect malware, regardless of its source or purpose. Politics don't even enter the discussion, nor should they.
Except, they just entered the discussion a half second ago when you tried to justify your failure and explain it away on the back of a political rationale.
The truth is, consumer-grade antivirus products can't protect against targeted malware created by well-resourced nation-states with bulging budgets.
No, the truth is, consumer-grade antivirus can't protect against garden variety malware, but it can apparently detect the well-resourced nation-state malware. Oh, that makes me wonder, what do you offer that is better than consumer-grade? Other than a bigger price tag, does it do a better job detecting malware?
But targeted attacks like these go to great lengths to avoid antivirus products on purpose.
Once again, this implies regular malware doesn't care about being detected. DERP #2?
And the zero-day exploits used in these attacks are unknown to antivirus companies by definition.
And again, this is bullshit. Stuxnet had 3 known vulnerabilities (CVEs 2010-2568, 2010-3888/2010-3338, 2010-2729), 4 if you count the default credentials in SIMATIC that it could leverage (CVE 2010-2772). Flame apparently has 2 known vulnerabilities (CVEs 2010-2568, 2010-2729). Even worse, if antivirus companies had paid attention to these samples sitting in their archives, they may have ferreted out the vulnerabilities before they were eventually disclosed.
As far as we can tell, before releasing their malicious codes to attack victims, the attackers tested them against all of the relevant antivirus products on the market to make sure that the malware wouldn't be detected.
Way to spin more Mikko. This is standard operating procedure for many malware writers. I believe VirusTotal is the first stop for many bad guys.
They have unlimited time to perfect their attacks.
This is patently absurd. If Stuxnet was designed by western intelligence to hinder Iran's nuclear program, then they most certainly didn't have "unlimited time". They were on a clock to write it and get it released in a manner that would help ensure it reached the goal. That had to be done before Iran had a fully mature nuclear program. DERP #3 Mikko.
It's not a fair war between the attackers and the defenders when the attackers have access to our weapons.
Last I checked, you have access to the largest arsenal of attacker's weapons ever. Further, you have access to some malware writers that turned legitimate. This is not a valid excuse for missing anything.
Flame was a failure for the antivirus industry. We really should have been able to do better. But we didn't. We were out of our league, in our own game.
That is because the antivirus industry created the game, and plays by different rules than the malware authors. The biggest difference? They have an end game, you do not. Antivirus is profitable because of the yearly subscriptions for new malware signatures. This is evident by the fact that PC consumers get antivirus software free with their computer. At trade shows, some vendors hand out their software for free, with a 30 day trial to updated signatures. That is the proverbial sample, to hook the kids on the good drugs, so they keep coming back for more and more.
When a big malware event makes the news, it only helps you. Antivirus firms are the first to jump on the bandwagon claiming that more people need more antivirus software. They are the first to cry out that if everyone had their software, the computing world would be a much safer place. The reality? Even computers with antivirus software get popped by malware. They don't detect all of those banking trojans and email worms like you claimed in this article. They don't protect against the constant onslaught of new threats. Your industry, quite simply, has no reason to improve their detection routines.
I remember in the early 90's, the next big thing in the land of viruses was polymorphing virus code. Antivirus vendors were developing "heuristics" that would detect this polymorphing code. In almost 20 years, how has the development of that gone? Obviously not stellar. Antivirus companies spend their time cataloging signatures of known malware, because that sells. "We detect 40 squirrelzillion viruses, buy our software!" What happened if you developed reliable heuristics and marketed it? "Buy our software, once, and the advanced heuristics can catch just about any malware!" There goes your business model. So of course you don't want to evolve, you want to wallow in your big pile of shit because it is warm and comfortable. You can ignore that overwhelming smell of shit that comes with your industry, because of the money. Don't believe me? Let's hear it from the pro:
There's still job security in security, but it doesn't take away the fact that I'm disappointed about how poorly we've fared with this. -- Mikko Hypponen (May 29, 2012)
A couple Twitter denizens, namely @ochsff has challenged me on the point "just because you have a sample in a repository, you don't detect / flag it already." At face value, that is a good point and I understand it. However, consider two things. First, the article specifically says "[The Flame samples] had come through automated reporting mechanisms but had never been flagged by the system as something we should examine closely." Note automated reporting mechanism, implying that is not VirusTotal or F-Secure's Sample Analysis System (as @dlitchfield suggested), as those require manual interaction and the desire to upload a sample. Second, the article is sufficiently vague on which system it was obtained through. For example, did it come from VirusTotal who may share samples in an "automated" fashion to F-Secure? Did it come from a honeypot that uploads ANY binary added to the machine after a given date? Was it found via scraping malware forums or web sites looking for specific text strings to imply it may be new malware? Did it come from another person's existing collection that is automatically scraped every day? Without knowing, all we can do is guess how F-Secure got it. Before you condemn me for that, consider the second part of this; "Researchers at other antivirus firms have found evidence that they received samples of the malware even earlier than this, indicating that the malware was older than 2010." Sure seems to me like AV companies were detecting it in some fashion.
Update #2, June 3:Mikko has responded twice [one, two] now, saying that the sample F-Secure received was via VirusTotal, and it was uploaded by an Anonymous person in 2010. His second Tweet begins to make excuses again, saying that "We get 1000s of unknown samples via Virustotal every day. Some of them are clean, some are not; anybody can upload anything." First, someone uploading Flame in 2010 is very intriguing to me, because it begins to betray the idea that it was created by a "Western intelligence agency". If they spent hundreds of thousands of dollars developing this malware, would they really go and immediately hand it over to a site that shares samples with AV vendors? Why risk that it will get noticed and a signature developed before it has a chance to do its job? Second, and more to the point of this rant in the first place, is why it was uploaded and ignored for almost 2 years. To me, that seems like a failure on the AV industry for sure. I understand they are flooded with samples, but that also means they have a process in place to weed through them in some manner. They have to be able to pick some that will get signatures written for their products. The real breakdown, then, is in that process.
I am not an antivirus guy; I do not work in that part of the security industry. @peteconnolly asks what the solution is, if AV is totally useless. I don't think it is useless, but as @wimremes explained, it is more about knowing the limitations. In another thread, @markruots gives Mikko kudos for the apology, which I do too. I just wish it wasn't full of caveats and excuses, and wish it addressed the issues better. That is where I get into the rant about the AV not having incentive to change much. How can we fix it? Smart people like Mikko should be telling us that, not me. He is intimately familiar with the AV world and is likely one of the foremost experts on the industry as a whole. Where is his Wired article telling consumers that AV is broken, along with his suggestions on how they can fix it? As @mariovilas said, I am *not* familiar with the technology to any real depth. I understand the very high level process involved, but not the dirty details. I welcome him or anyone else to write more about the topic, so that the general public (me included) have a better glimpse into that world. I do not believe that anything uploaded to VirusTotal should be assumed to be malware. I do think more time should be spent figuring out generic tests that can identify false positives vs matches vs samples that should be studied more. Antivirus software has to use a scoring / weighting system of sorts these days, I think that system should be examined and more time spent on that than just blindly updating a huge signature database (unless that can be done a lot more efficiently).
Last, remember that my rebuttal is to the Wired article. I am replying to the information Mikko chose to share in that article. His choice of words (e.g., sample submitted "automatically") can be very prejudicial and lead to understandable assumptions. As always, I welcome him, F-Secure, or anyone reading this to chime in with more information. While this is a flamish rebuttal, it is not a one-sided slamfest in which I have no intention of holding on to my beliefs in a silly manner. If Twitter is too limiting, send me mail and let me know if I can quote/share from it in updates to this article.
Update #3, June 3:Looks like @imaguid has written the first lengthy correction / rebuttal to my own, and I sincerely appreciate the time and explanation. I encourage everyone to read it to better understand the AV world, something Mikko did not share, and that I did not have knowledge of. One thing I "got wrong" was due to the language Mikko used. When he said that the Flame sample had come through "automated reporting mechanisms", it sounded like something out there detected it, flagged it as interesting, and kept a copy for whatever reason (I presumed examination as time permitted). This in turn led to several things I said that were building on top of it. Now that Mikko has clarified that it was sent in an automated fashion from VirtusTotal to F-Secure, it makes more sense.
Regarding the 0-day angle, if it is not something AV can or should be concerned about, it shouldn't have been brought up. You are right, I meant the AV software should have detected the exploit attempts, not vulnerabilities. I also realize that is jumping ahead of what AV traditionally does, as it is not an IDS and generally not coded to detect exploits. However, moving forward, if these advanced worms are going to be all the rage, perhaps the AV industry should look at that.
Regarding the dates, the ones listed in that screenshot are the formal disclosure with enough details to identify it. Take the first for example, OSVDB 66387 that we have listed for 2010-07-16. F-Secure has a blog from a day before that in which they quote a Microsoft person saying they are investigating and will take appropriate action. There is no way Microsoft did that overnight, so the quote is old and/or Microsoft had exploit information before that. This F-Secure PDF says that a company detected a rootkit using the exploit almost a month earlier (2010-06-17). Check the Microsoft advisory and we see at least 5 people worked with Microsoft on this. It is basically a given that multiple people had this exploit or vulnerability information far before 2010-07-16. In the world of VDBs, we see many of the big vulnerabilities disclosed to a vendor by multiple parties (e.g., ZDI, iDefense, other security companies). It is illogical to think that only the 'good guys' have these vulnerabilities. It is rational to assume that as many 'bad guys' have it as good. When Stuxnet was coded, they were likely 0-day vulnerabilities to the people who created the malware. But, it is also probable that other people had figured out the same exploits, both good and bad.
Update #4, June 4:The Defcon 406 blog has an interesting rebuttal to Mikko that asks who he is apologizing to, and points out that it is essentially a marketing piece; "You're not being altruistic, you're bein opportunistic."
