Getting The Gist Of Things
This blog post represents a progression of thought. I'm not making arguments, I'm making suggestions as they occur to me.
Security is very hard to quantify- it's the classic "how to prove a negative" problem, coupled with the "every application has bugs" problem, and wrapped up with the "holy crap this is impossible" problem. Maybe we don't need to accurately quantify it though. I mean, when we're doing the risk assessment part, we want to be close, but as far as the remediation phase? In my experience, at the end of a vulnerability assessment, people usually care about about three things:
a) What issues do I need to fix?
b) What kinds of issues are most common?
c) Where does my organization stand (relative to the competition and the global web application environment)?
When people focus on one of these issues, which one they fixate on can be very telling. While somebody should always take a look at the report and fix the specific issues (where possible), the people who only care about the found vulnerabilities often refuse to look at the big picture.
If we're doing vulnerability assessments to find bugs, we're going to fail. This is the problem with every assessment: It won't find all the vulnerabilities. What it can do is find "a bunch" of vulnerabilities, which collectively can inform you about where you need to put your efforts. The job isn't done when you've patched the bugs. The goal of a vulnerability assessment, at least from an action standpoint, is to know where you stand security-wise. If you can do that, you can manage the specifics.
In Jeremiah's response to my last blog post, he discussed some of the problems with web app scanner inconsistencies, making the point that all scanners will miss something, and that some are better than others at specific things. I'm going to neatly ignore that and talk about something slightly different.
He's right, but it's a limited view of the problem. A scanner may not be able to find a specific issue, or all instances of a specific issue, but a human tester won't either. The quick wins that the scanner provides in tracking down holes should be considered a byproduct- we should use the scanner primarily to get a feel for where the problems exist. If you find out that your servers are rarely up-to-date, or that you have a high concentration of XSS vulnerabilities, that is useful information in itself. The most important part of a report should be a venn diagram of weak spots, not a list of vulnerabilities. I'd argue that, from a high-level security management perspective, whether an exploitable issue is actually found or not is irrelevant. That's not the goal.
The scanner's results will be skewed and inconsistent, just like the human tester. Unlike the human tester, a scanner will be consistently inconsistent. We can compensate for that. If we have base metrics regarding which vulnerabilities are most common in which situations (numbers we're only beginning to put together as a community), and we know that a scanner consistently produces skewed results, we can calibrate our reports to compensate. We can then spot check the weak points to confirm our findings, and further calibrate the reporting process.
Accuracy is always ideal, but lacking that, the ability to make reasonable estimates about where the problems lie would speed up the bug hunting process, the patching process, and allow us to focus training, policy and development procedures. With that approach, the problem of scanners "missing things" largely goes away, and in fact even the worst scanner will still produce useful results. On top of that, scanners can be easily compared with simple relevancy ratings between the predicted vulnerability distribution, the discovered issues, and actual findings (from supplementing the scanner with human testing).
Scanners are a tool in the bag, and they're a pretty good one. If the results aren't interpreted correctly, it doesn't matter if you have the world's best bug hunter on your side. An assessor's most important role isn't in finding the bugs- it's in interpreting the results.
Security is very hard to quantify- it's the classic "how to prove a negative" problem, coupled with the "every application has bugs" problem, and wrapped up with the "holy crap this is impossible" problem. Maybe we don't need to accurately quantify it though. I mean, when we're doing the risk assessment part, we want to be close, but as far as the remediation phase? In my experience, at the end of a vulnerability assessment, people usually care about about three things:
a) What issues do I need to fix?
b) What kinds of issues are most common?
c) Where does my organization stand (relative to the competition and the global web application environment)?
When people focus on one of these issues, which one they fixate on can be very telling. While somebody should always take a look at the report and fix the specific issues (where possible), the people who only care about the found vulnerabilities often refuse to look at the big picture.
If we're doing vulnerability assessments to find bugs, we're going to fail. This is the problem with every assessment: It won't find all the vulnerabilities. What it can do is find "a bunch" of vulnerabilities, which collectively can inform you about where you need to put your efforts. The job isn't done when you've patched the bugs. The goal of a vulnerability assessment, at least from an action standpoint, is to know where you stand security-wise. If you can do that, you can manage the specifics.
In Jeremiah's response to my last blog post, he discussed some of the problems with web app scanner inconsistencies, making the point that all scanners will miss something, and that some are better than others at specific things. I'm going to neatly ignore that and talk about something slightly different.
He's right, but it's a limited view of the problem. A scanner may not be able to find a specific issue, or all instances of a specific issue, but a human tester won't either. The quick wins that the scanner provides in tracking down holes should be considered a byproduct- we should use the scanner primarily to get a feel for where the problems exist. If you find out that your servers are rarely up-to-date, or that you have a high concentration of XSS vulnerabilities, that is useful information in itself. The most important part of a report should be a venn diagram of weak spots, not a list of vulnerabilities. I'd argue that, from a high-level security management perspective, whether an exploitable issue is actually found or not is irrelevant. That's not the goal.
The scanner's results will be skewed and inconsistent, just like the human tester. Unlike the human tester, a scanner will be consistently inconsistent. We can compensate for that. If we have base metrics regarding which vulnerabilities are most common in which situations (numbers we're only beginning to put together as a community), and we know that a scanner consistently produces skewed results, we can calibrate our reports to compensate. We can then spot check the weak points to confirm our findings, and further calibrate the reporting process.
Accuracy is always ideal, but lacking that, the ability to make reasonable estimates about where the problems lie would speed up the bug hunting process, the patching process, and allow us to focus training, policy and development procedures. With that approach, the problem of scanners "missing things" largely goes away, and in fact even the worst scanner will still produce useful results. On top of that, scanners can be easily compared with simple relevancy ratings between the predicted vulnerability distribution, the discovered issues, and actual findings (from supplementing the scanner with human testing).
Scanners are a tool in the bag, and they're a pretty good one. If the results aren't interpreted correctly, it doesn't matter if you have the world's best bug hunter on your side. An assessor's most important role isn't in finding the bugs- it's in interpreting the results.
Labels: Scanning
posted by mckt at
5:29 PM
0 Comments:
Post a Comment
Subscribe to Post Comments [Atom]
<< Home