When Should Organizations Use Penetration Tests?

TL;DR

Penetration testing delivers the most value after an organization has established its foundational security controls and can remediate the findings. For less mature programs, the same budget and effort may be better spent assessing control maturity, testing people and response processes, simulating common breach events, and directly helping teams fix systemic weaknesses.

Transcript

Hello. I'm Adrian Sanabria, Director of Research for ThreatCare, and I want to welcome you to my session for RSA Conference 2018, It's Time to Kill the Pen Test. This is an RSA quick look at what I'll be covering. So killing the pen test, what am I talking about here? Understanding the relationship between pen testing and the basics is really impor... Read More

Key Insights

• Penetration testing skills remain valuable, but the packaged assessment commonly sold as a penetration test should often be replaced. The argument targets low-value engagements, not the broader practice of applying offensive techniques to verify controls, identify vulnerabilities, and determine whether weaknesses can be exploited.
• Most penetration tests are driven by compliance checkboxes and suffer from inconsistent definitions and quality. Different providers include different tasks and expect different skills, so organizations purchasing a penetration test may receive services ranging from little more than scanning to meaningful exploitation and control validation.
• A typical penetration test spends much of its effort running vulnerability scanners and producing a report. Only about twenty percent of the described engagement represents actual penetration testing, including validating scanner findings, attempting exploitation, and pivoting after a successful compromise.
• Experienced testers can often predict the findings after a short discussion about an organization's environment and infrastructure. When basic weaknesses are obvious, a lengthy exploitation exercise adds little information and may direct scarce remediation effort toward issues that are not the organization's highest priorities.
• Penetration tests work best when foundational security controls are already established and the organization feels confident in its program. At that stage, testing can meaningfully evaluate prior security investments without merely producing a noisy, prioritized list of basic vulnerabilities.
• Many organizations lack the maturity, staffing, or skills required to act on penetration-test results. Remediation can take weeks, months, or years, and repeated annual engagements may uncover the same vulnerabilities or even artifacts left by previous testers.
• The CIS Top 20 places penetration testing last, which the speaker interprets as a meaningful ordering. Organizations should address the preceding foundational controls before using a penetration test as a higher-maturity evaluation of how effectively their established security program performs.
• A replacement assessment should evaluate security-program maturity, test existing controls, examine people and response processes, and simulate common breach events. Automation and sampling can identify systemic failures quickly, preserving more engagement time for helping the organization repair weaknesses directly.

Questions & Answers

Q: When should an organization conduct a penetration test?

An organization should conduct a penetration test after it has established most foundational security controls, feels reasonably confident in its security program, and has the staff and skills to address discovered weaknesses. At that maturity level, the assessment can test whether prior security work performs as intended instead of producing a noisy list of basic, predictable vulnerabilities.

Q: Why do many penetration tests provide limited value?

Many penetration tests provide limited value because they are driven by compliance checkboxes, vary widely in scope and quality, and devote substantial effort to scanning and report production. Experienced testers can often anticipate the findings from a brief discussion of the environment. If an organization cannot remediate those findings, the assessment may repeat known problems without improving security.

Q: What is the difference between penetration testing and a penetration-test engagement?

Penetration testing is a skill set used to test security controls, determine whether systems are vulnerable, and verify whether weaknesses can be exploited. A penetration-test engagement is the packaged assessment performed internally or purchased from a provider. The proposed change concerns replacing many low-value assessment packages, not eliminating the underlying offensive security skills used for meaningful validation.

Q: How is a typical penetration-test engagement inefficient?

A typical engagement may begin with a tester connecting a machine and running a vulnerability scanner. The tester then validates selected results, attempts exploitation, and may pivot from successful compromises. In the example described, only about twenty percent of the work is actual penetration testing, while considerable time is spent writing findings in Microsoft Word and converting the report into a PDF.

Q: Why can penetration testing do more harm than good?

Penetration testing can do more harm than good when it consumes limited money and staff attention while producing predictable findings that the organization cannot address. It may create work that is not the highest priority, while foundational program weaknesses remain unresolved. Some organizations undergo repeated tests for years and continue receiving the same findings because their remediation capacity is inadequate.

Q: What should replace penetration tests for less mature organizations?

Less mature organizations should use an assessment that examines the security program's overall maturity, evaluates existing controls, tests whether detection and response processes work, and simulates common breach events. The engagement should minimize time spent trying to break systems and maximize time spent helping the client correct identified weaknesses and improve security directly.

Q: How can security assessments use automation and sampling?

Automation and sampling can shorten the discovery phase by revealing systemic problems without checking every system individually. If a sample finds five or six systems without antivirus, or discovers Java that is three years out of date, the assessor already has evidence of broader control failures. The remaining engagement time can then focus on remediation and program improvement.

Q: How can an assessment test detection and response capabilities?

An assessment can simulate common breach events and observe whether controls, processes, and people identify and handle them. The transcript suggests testing data-loss controls by placing fake but apparently valid credit card numbers in a text file or spreadsheet, then uploading or emailing the file. The exercise checks whether alerts are generated, noticed, and acted upon.

Summary & Key Takeaways

• Most commercial penetration tests provide limited value because checkbox requirements have reduced quality and created inefficient engagements. A typical assessment relies heavily on vulnerability scanning, reserves only a fraction of the effort for exploitation, and consumes substantial time producing reports. Organizations may already know the likely findings before testing begins in practice.

• Penetration testing is most useful when an organization already has a mature security program, functioning controls, and the capacity to remediate findings. Many organizations have not reached that stage. Some repeatedly receive the same findings over multiple years because remediation takes weeks, months, or years, leaving foundational weaknesses unresolved and unchanged.

• A better assessment would minimize time spent breaking systems and maximize direct security improvement. It would evaluate program maturity, verify existing controls, test detection and response, and simulate common breach events. Automation and sampling can expose systemic problems efficiently, allowing consultants to spend more of the engagement helping the client correct underlying weaknesses.