How to Measure Security Program ROI Effectively

TL;DR

Security investments should be judged by clear business objectives, measurable success criteria, total resource costs, and opportunity costs. More products or recurring assessments do not automatically improve security, especially when results lack context. Effective programs test the right risks at an appropriate depth, document what was actually examined, and reconsider activities whose value declines over time.

Transcript

Thank you. So my name is Window Snyder, and I have been in this security space for over twenty years. And I've had a chance to work in a lot of different kinds of security organizations. I've worked at big companies like Microsoft and Apple, and small companies, some startups, and open companies like Mozilla, and, uh, a really diverse set of securi... Read More

Key Insights

• A security program is effective only when it has a clear objective, a genuine business need, and defined criteria for success. Activities performed merely because they are customary, recommended by a book, or embedded in the development lifecycle may not remain worthwhile.
• Red teaming is better aligned with evaluating an organization’s response than with simply discovering whether an outside attacker can enter a network. It may also support staff enjoyment, but the intended benefit should be explicit so the program can be evaluated against that purpose.
• The full cost of a security change includes the time required from everyone affected by it. Introducing multi-factor authentication or behavioral training can create high-touch support needs and force employees across the organization to learn new processes, increasing costs beyond deployment.
• Opportunity cost is part of security program evaluation because people assigned to one activity cannot address another priority simultaneously. A program that once met a business need may become a weaker investment if circumstances change or more important work emerges.
• Application security testing produces diminishing returns as reviewers become familiar with a system, identify vulnerabilities, and eventually find fewer new issues. Teams commonly stop internal reviews at that trailing stage, but they may apply less discipline when deciding the length or recurrence of vendor engagements.
• A vendor’s failure to find vulnerabilities does not establish that software is robust or resilient. The outcome could reflect comprehensive testing, insufficient depth, unfamiliarity with the code, an unsuitable methodology, or differences in the experience of the people assigned to the engagement.
• Vendor assessment value is easier to judge when the buyer receives evidence about what was attempted, which components received attention, how time was allocated, and what methodology was used. These details may not be part of the vendor’s default deliverable and should be requested.
• Repeated network penetration testing can remain useful because networks and vulnerability knowledge change over time. New hosts may appear, machines may fall behind on updates, and tools may gain new signatures, allowing the same general testing operation to produce meaningfully different results.

Questions & Answers

Q: How do you measure the ROI of a security program?

Measure security program ROI by first defining the program’s objective and the business need it serves. Establish criteria that indicate success, identify every resource the program consumes, and compare its benefits with other work the organization could perform. Revisit the decision over time because a program started for a valid reason may no longer be the best use of people, money, or attention.

Q: What costs should a security investment include?

A security investment should include more than the purchase price of software, vendor fees, and the staff directly responsible for deployment. It should also include time spent by employees who must learn changed processes, seek support, complete training, or modify their behavior. Organization-wide changes such as multi-factor authentication can therefore cost substantially more effort than the initial implementation alone suggests.

Q: When is red teaming a useful security activity?

Red teaming is useful when its objective matches what the exercise can meaningfully evaluate. Testing the organization’s response to an attempted compromise is presented as a strong reason for conducting it. Using red teaming mainly to discover whether an outside attacker can enter the network may be less effective. Staff enjoyment can also be a legitimate benefit if it is acknowledged and assessed deliberately.

Q: Why should security teams consider opportunity cost?

Security teams should consider opportunity cost because every recurring program uses people and time that could address another risk or business priority. An activity may continue simply because the organization has always performed it, even when the original need has changed. Evaluation should therefore ask whether something more important exists for the team or affected employees to work on instead.

Q: When should application security testing stop?

Application security testing should be reconsidered when additional effort produces progressively fewer vulnerability findings. Reviewers initially spend time learning the system, then become productive, and eventually reach a trailing stage of diminishing returns. The stopping point should reflect the application’s importance, the depth already achieved, and whether further effort is more valuable than moving to another codebase, product, or security priority.

Q: Does a clean penetration test prove software is secure?

A clean penetration test does not prove that software is robust or resilient. Testers may have conducted a comprehensive review and found nothing significant, but they may also have lacked sufficient time, expertise, methodological coverage, or familiarity with the codebase. Without evidence describing what was attempted and how deeply each component was examined, the absence of findings remains difficult to interpret.

Q: What should a security vendor report after testing?

A useful vendor report should explain what the testers actually tried, which components they examined, how much time they spent on different areas, and what methodology guided the work. These details help the buyer distinguish a thorough assessment from a shallow or poorly matched engagement. Because such information may not be included in the default deliverable, the customer should establish expectations through the vendor relationship.

Q: Why can repeated network penetration testing remain valuable?

Repeated network penetration testing can remain valuable because the tested environment and available detection knowledge may change. New hosts can be added, previously updated machines can become outdated, new vulnerabilities can be identified, and testing tools can acquire new signatures. Success can be evaluated more concretely by confirming that a defined set of vulnerability checks was performed against a defined set of IP addresses.

Summary & Key Takeaways

• Security programs need explicit objectives tied to business needs, along with criteria that show whether they work. Activities such as red teaming, certifications, conferences, training, and product assessments may have value, but their purpose must be stated and measured instead of being justified by habit or inclusion in a standard process.

• The true cost of a security initiative includes more than software, vendors, and deployment staff. Changes such as multi-factor authentication or behavioral training can consume time throughout an organization. Leaders should therefore consider total resource use, opportunity cost, continuing business relevance, and whether security engineers find the work engaging and worthwhile.

• Security testing often produces diminishing returns as reviewers learn a system, find vulnerabilities, and eventually discover fewer issues. Vendor reports require additional scrutiny because finding nothing does not prove resilience. Buyers should request details about methods, time allocation, components examined, personnel, and testing depth before deciding whether an engagement delivered meaningful value.