A Penetration Test Is Not A Vulnerability Assessment

There's been a bit of debate going around lately, in which David Maynor said that we should be using 0-day exploits in pentests, and subsequently got yelled at for it. Just for the sake of taking sides, I feel like I should point out that anybody who says 0-days don't belong in pentests have no idea what a penetration test is.

This isn't all that uncommon, so let's have a little vocabulary lesson:

Security Audit A technical assessment of an environment's security. This is a catchall term that may include one or more of the components below as part of a full security program.

Penetration Test An evaluation of security controls through a simulated attack. This can be automated (as real attacks can be), though it certainly will not be effective as a manual attack by a skilled attacker (as real attacks also can be). The goal: to evaulate the effectiveness of security controls, including incident response. Vulnerabilities may be found, but that is a side effect of the process, and a secondary goal at best.

Vulnerability Assessment A review of applications, configurations, and systems for vulnerabilities such as bugs, misconfigurations, or architectural flaws. This also may be automated (and should be, for efficiency) as well as manual (and should be, for thoughness). The goal: to find vulnerabilities, presumably for remediation, although vuln metrics can be useful in other ways, such as quality assurance and developer skills evaluation.

Risk Assessment Quantification and prioritization of assets and vulnerabilities. The goal: To get numbers about security posture and to focus remediation efforts where they are most needed.

While it may sound like I'm picking nits, each of these processes has distinct goals-- they gather different metrics, they create different kinds of data, and they are used for different purposes. There's certainly some overlap, and there are other compoments to security auditing, but these are the ones I see most often confused.

There's a lot of reasons people mix these terms up- Sales drones in this industry are often confused about the goals, the processes, and the results. Some people deliberately blur the lines for compliance reasons (For example, PCI requires anual penetration tests. By blurring the lines, many organizations get away with fulfilling this requirement without actually running a penetration test. I'll leave the question of whose fault this is, and whether it is a bad thing results-wise up to the reader). Salespeople, technicians, and management alike often use the word "Penetration Test" just because it's sexier. Regardless of why, the processes are not the same.

When I begin work on a "penetration test" only to find out that the client is expecting a risk assessment and the sales department sold (and scoped) the project as a vulnerability assessment, it's not just frustrating, it's dangerous. Confusion about these terms leads to poorly written contracts, misunderstood objectives, and miscommunication about rules of engagement. Let's all try to be a bit more careful about what we're saying.

Coming back to the original point of contention, it's obvious that penetration tests should include discovery and exploitation of 0-day vulnerabilities if possible. You're trying to simulate a real attack. Real attackers do things like that.

If all you're doing is running a few automated scanners to find vulnerabilities, you're not performing a penetration test. You're performing a vulnerability assessment. There's a place for that. I'd even argue that the vuln assessment is a more important part of a complete security program, as long as management, development, and compliance people are already on board with this whole "security" thing.

Please stop using words wrong. We're a young industry, and we don't want to get stuck with incorrect definitions and unclear vocabulary.