How to Fix the ASVs

McAfee has been hit very hard this week, and the vulnerabilities in the other ASVs that I published are starting to make waves as well. While it's fun (and it really is) to point a finger and say "LOL! McAfee Fail!", this really isn't helpful. It's time we take this as an opportunity to look at things that are seriously wrong with PCI and the way that it is implemented.

I don't claim to be an expert on the PCI standards. I am just one of many people who is trying to improve the security of his company and the internet as a whole. In my view, achieving compliance should be an afterthought. While any business executive worth his paycheck should understand and value security from a risk-management perspective, the fact is that most respond better to industry requirements. Regulation is a dirty hack, but it at least gets the attention it needs.

Since the service providers already understand that they need to comply with these regulations, we may as well fix them so that they are effective. In my view, here are the major changes that need to be made:

Better Education
For many small service providers, the PCI testing process is the first time they look seriously at their own security. Having them sign a few forms and running a scan on their servers does not educate them, and it does not prepare them to deal with security properly. This sets a poor precedent for a growing company, and in my experience, it invariably causes problems down the road. Having an expert consultant educate the service provider about how to design a security program would do more to protect cardholder, customer and company data than any set of prescriptive security regulations. This phase of the process is usually skimmed over by a company trying to get the not-so-elusive compliance certificate. In my opinion, it may be the most important part.

Better documentation
Without clear definitions of the standards and recommendations for compliance, we'll never get far. Most service providers do not even have a clear understanding of which standards they are expected to comply with, much less how to do so. Instead, they speak with the sales department at the cheapest ASV they can find, perform a scan, and send the vulnerability reports to their system administrators.

To be fair, there is documentation out there, but there's a lot of misinformation too. The official PCI FAQ (which is itself vulnerable to XSS) doesn't provide much useful information beyond the standards themselves and a few supporting documents. Several unofficial forums have sprung up, but the information contained within isn't always reliable.

More Responsibility
To my knowledge, this week's McAfee issues, as well as the issues with other ASVs, have not affected their status as a PCI scanning vendor at all. Given that some of the ASVs' XSS holes have still not been fixed, I find this extremely shocking. PCI companies need to treat customer data responsibly, and need to be held responsible when they do not. Revoking their ability to perform PCI scans would be disastrous to their business, but the stated goal of the PCI standards isn't to keep poorly managed companies in business. If the true goal of these standards is to "help organizations proactively protect customer account data", then the ASVs need to be held responsible when they do not.

PCI ASVs should be held to much more rigorous standards than the Service Providers they certify. I was shocked to find out that regular third-party penetration tests are not a requirement for ASVs. From what I understand (and I may be misinformed), they essentially fill out paperwork and assure the PCI board that they are themselves compliant. Does this strike anybody else as insane?

More Transparency
While the PCI Council provides documentation of proper scanning procedures, they do not go into much depth. In particular, they barely touch on web application scans, while in my biased opinion, this is one of the most vulnerable and therefore critical parts of most Service Providers' systems.

No data is available to the public regarding the relative efficacy of the scans. I'm not aware of anybody with a truly effective web app scan, but I've run several on the same servers (in a rather unscientific fashion), and come up with vastly different results. The free PCI scan offered by nCircle took less than an hour, and only did a cursory inspection of my web application. By contrast, McAfee's scan took close to 3 hours, most of which was spent testing the web application. Neither found the rather basic XSS and SQLi holes that I deliberately put in.

While neither app was up to par, the McAfee scan was clearly superior- at least at finding web application holes. At the time, I was more interested in finding problems with the scanning services, rather than doing a direct comparison of the two. However, it would not be difficult to design a system for measuring the effectiveness of ASV scans. The PCI council should do so. They should make the results of these tests public, and should make scanners' status as an ASV dependent upon them.

Better Enforcement
An end user has a right to know whether the company he is doing business with is PCI compliant or not. A verification process, even as simple as a list of compliant companies on the PCI website, is essential to an effective certification program. Of course, when problems are found, the Service Provider in question should have their certification revoked.

I have come across many companies with poor security practices, and many simply do not care. The financial institution that I bank with used to have its access logs publicly accessible. I first alerted them to the issue over a year ago, and no action was taken. In the course of my PCI research, I coincidentally found out who their auditor was, and after alerting them to the issue, the logs were removed almost immediately.

Ideally, I should not be able to find out who performs the auditing on a company (lest I perform attacks through the ASV). However, only by reporting this issue to the auditor was I able to get it fixed. If I were able to report it directly to the PCI board, and a system were in place for getting that report to the right people, this would not have been a problem.

Better Testing
Finally, we arrive at the elephant in the room: the testing performed by the ASVs sucks.

Scanning has a place in security- it can find unpatched software, poor configuration, and other issues that often go unnoticed. However, until we have strong AI in the scanners, it will never have the ability to detect logic flaws or reliably fuzz for application vulnerabilities.

James Lester (Senior Security Analyst for McAfee, of all places) touched on this last month, and I think he's right- scanning is not effective in and of itself, and should not be relied on as the sole method of testing for PCI compliance. A full-scale audit will find issues that no scanner can, and more importantly, it will give auditors a reliable "feel" for the security of a Service Provider. Any provider that builds their own applications for payment and customer data management should be required to perform a QSA audit, regardless of the scale of the application.

Those Service Providers that cannot afford a full audit should not be allowed to build their own applications. Instead, they should use one of the already approved applications, on a sandboxed or private system. Scanning should still be required, but this way we can limit the scope of this scan to finding out-of-date software, unpatched holes, and configuration issues- things that the scanners are already very good at.