Getting The Gist Of Things

This blog post represents a progression of thought. I'm not making arguments, I'm making suggestions as they occur to me.

Security is very hard to quantify- it's the classic "how to prove a negative" problem, coupled with the "every application has bugs" problem, and wrapped up with the "holy crap this is impossible" problem. Maybe we don't need to accurately quantify it though. I mean, when we're doing the risk assessment part, we want to be close, but as far as the remediation phase? In my experience, at the end of a vulnerability assessment, people usually care about about three things:

a) What issues do I need to fix?

b) What kinds of issues are most common?

c) Where does my organization stand (relative to the competition and the global web application environment)?

When people focus on one of these issues, which one they fixate on can be very telling. While somebody should always take a look at the report and fix the specific issues (where possible), the people who only care about the found vulnerabilities often refuse to look at the big picture.

If we're doing vulnerability assessments to find bugs, we're going to fail. This is the problem with every assessment: It won't find all the vulnerabilities. What it can do is find "a bunch" of vulnerabilities, which collectively can inform you about where you need to put your efforts. The job isn't done when you've patched the bugs. The goal of a vulnerability assessment, at least from an action standpoint, is to know where you stand security-wise. If you can do that, you can manage the specifics.

In Jeremiah's response to my last blog post, he discussed some of the problems with web app scanner inconsistencies, making the point that all scanners will miss something, and that some are better than others at specific things. I'm going to neatly ignore that and talk about something slightly different.

He's right, but it's a limited view of the problem. A scanner may not be able to find a specific issue, or all instances of a specific issue, but a human tester won't either. The quick wins that the scanner provides in tracking down holes should be considered a byproduct- we should use the scanner primarily to get a feel for where the problems exist. If you find out that your servers are rarely up-to-date, or that you have a high concentration of XSS vulnerabilities, that is useful information in itself. The most important part of a report should be a venn diagram of weak spots, not a list of vulnerabilities. I'd argue that, from a high-level security management perspective, whether an exploitable issue is actually found or not is irrelevant. That's not the goal.

The scanner's results will be skewed and inconsistent, just like the human tester. Unlike the human tester, a scanner will be consistently inconsistent. We can compensate for that. If we have base metrics regarding which vulnerabilities are most common in which situations (numbers we're only beginning to put together as a community), and we know that a scanner consistently produces skewed results, we can calibrate our reports to compensate. We can then spot check the weak points to confirm our findings, and further calibrate the reporting process.

Accuracy is always ideal, but lacking that, the ability to make reasonable estimates about where the problems lie would speed up the bug hunting process, the patching process, and allow us to focus training, policy and development procedures. With that approach, the problem of scanners "missing things" largely goes away, and in fact even the worst scanner will still produce useful results. On top of that, scanners can be easily compared with simple relevancy ratings between the predicted vulnerability distribution, the discovered issues, and actual findings (from supplementing the scanner with human testing).

Scanners are a tool in the bag, and they're a pretty good one. If the results aren't interpreted correctly, it doesn't matter if you have the world's best bug hunter on your side. An assessor's most important role isn't in finding the bugs- it's in interpreting the results.

Are Secure Web Apps Possible? Yes (and no)

Mike Rothman wrote a post addressing the general problem of web security, titled "Are Secure Web Apps Possible?". My answer: Yes. And no. The general conclusion Mike comes to is that web scanners are useful for generating quick wins, but hardly complete. As long as they are used with that knowledge, they are excellent tools to have available.

He's absolutely right, but there are some problems with web application scanners. Well, "some" is an understatement. In traditional application testing, the attack surface is relatively small, and fairly well-defined. The rules for building a secure application haven't changed much in the last few years, and the basic language communicating with the hardware has been relatively stable for quite some time. At the very least, it's well-documented. The fact that we still have critical bugs popping up in common applications has more to do with the development process than the technology itself.

With web applications, it's not so simple. The platform is not the hardware. The platform is the web browser, and the web browser speaks hundreds of dialects, in dozens of languages, and they all interact. There are many more layers between the user and the hardware, and every one of them is exposed- to the network, to malicious users, to malicious servers, etc. The attack surface that presents is massive, but more importantly, it's dynamic. With new standards being added to the browsers, old ones being phased out, and the way that they interact changing constantly, creating a scanner that understands the context a web site runs in seems impossible. When you consider that there are about a dozen browsers currently in mainstream use (including mobile versions), the problem just compounds.

I don't expect web scanners to ever be able to address those problems, but I hope I'm wrong. Currently, it takes at least a few years of dedicated experience before you really get a "feel" for finding and assessing bugs, and RSnake doesn't scale well.

My opinion is that the fundamental architecture of the web needs to change drastically before it is even remotely close to "secure." Until then, all we can do is fix bugs, fix development processes, fix user behavior, or try to hack on kludges when the problems come up.

The last solution doesn't seem very helpful, so let's take a look at the other three.

Fortunately, in many cases, the subtle and obscure bugs aren't the problem. Security isn't black and white, and the trivial ones are easy to fix. If a company like AT&T hasn't at least found the simple issues (like the ones used in the recent so-called "hack"), they are being negligent. Mike is right, getting a few quick wins with a web scanner isn't a bad idea- just make sure you know what that means. You won't get everything, but you'll get the easy stuff, and this is where risk assessments really shine. That's a much bigger topic than this blog post can handle though.

Fixing development processes is just starting to catch on. Web developers, browser developers, and low-level system developers alike are beginning to adopt SDLs, and we're already seeing positive results. People are developing frameworks for secure development (more on that in future posts). Things are happening.

If development processes are hard to change, user behavior is next to impossible. Still, the truly paranoid user can expect a decent level of security through careful system configuration, limited browser functionality and general security awareness.

The general conclusion, something we all know, is that security is an extremely difficult, complex process. Many facets are involved, but each can only be addressed by a specific party. It's each of their responsibility to help all of us.