How Can Security Metrics Reveal What Matters?

TL;DR

Security programs need reliable metrics to distinguish controls that materially improve outcomes from expensive activities that do not. By identifying security's degrees of freedom, organizations can replace superstition, precedent, and intuition with evidence-based decisions, much as baseball teams used detailed analytics to outperform better-funded competitors and expose previously accepted practices that contributed little value.

Transcript

Ladies and gentlemen, please welcome Hugh Thompson. Good morning. Welcome. Welcome to day two of RSA Conference. I hope everybody is having a good time here in Amsterdam. So I, uh, I got a little concerned last night after the flash talks presentation. I overheard a, a fairly sizable group of attendees talking about visiting some of the local coffe... Read More

Key Insights

• Information security lacks dependable metrics for proving whether a purchased system, training campaign, or established practice produces outcomes that matter. Without such evidence, organizations can continue funding activities based primarily on precedent, assumptions, and the expectation that a security product must be useful.
• A degree of freedom is an independent way a dynamic system can change without violating its constraints or affecting the relevant outcome. A garbage truck's color can vary without changing its function, while a flat tire directly interferes with its ability to collect garbage.
• The maturity of a field is reflected in how quickly it can distinguish meaningful variables from degrees of freedom. Security remains immature in this respect because practitioners often cannot determine which controls influence protection and which expensive activities contribute little to business security.
• Software testing was shaped by superstition when teams lacked evidence about which conditions caused failures. In the server crash story, testers examined the child's height, possible wire pulling, wet diaper, and movements before identifying the keyboard interaction that actually produced the crash.
• The server crash was caused by an interaction between Sticky Keys and a named pipe used by the application. The two-year-old sat on the keyboard and Shift key, activating Sticky Keys and revealing a defect that sustained testing by the development team had missed.
• Baseball previously relied heavily on scouts' impressions when making multimillion-dollar decisions about players. The Moneyball transition replaced much of that intuition with statistical analysis of performance, showing how measurement can challenge deeply established decision-making practices in a mature industry.
• Data-driven transitions can provoke fear because measurement may expose bad investments, threaten established roles, and show that respected practices do not materially affect outcomes. Security professionals may face the same discomfort when evidence reveals that familiar products or programs are degrees of freedom.
• Early adopters of baseball analytics achieved stronger results despite limited resources. Teams with small budgets reached the playoffs and outperformed competitors with budgets sometimes an order of magnitude greater, demonstrating that better measurement can matter more than simply spending more money.

Questions & Answers

Q: What are degrees of freedom in information security?

Degrees of freedom are variables that can change without affecting the relevant outcome or violating a system's constraints. In information security, the concept can be used to identify products, processes, training campaigns, or other activities that consume resources but do not materially improve security. Finding these variables requires dependable metrics that distinguish consequential controls from irrelevant variation.

Q: Why are metrics important for information security?

Metrics are important because security teams need evidence that their investments and practices produce meaningful results. Without reliable measurement, it is difficult to justify whether a box in a data center, a training campaign, or another costly measure actually improves security. Metrics provide ground truth for separating effective controls from activities maintained through superstition, precedent, or untested assumptions.

Q: How does the Sticky Keys story illustrate degrees of freedom?

The testing team investigated many variables after a two-year-old caused a server product to crash, including his height, possible contact with wires, wet diaper, and movements through the lab. Those factors did not matter. The relevant action was sitting on the keyboard and Shift key, which activated Sticky Keys and created a conflict with the application's named pipe.

Q: What caused the server product to crash during testing?

The crash resulted from an interaction between the Windows Sticky Keys feature and a named pipe used by the enterprise server application. The child sat on the keyboard and Shift key, activating Sticky Keys after repeated Shift input. That interaction produced the blue screen of death, revealing a defect that the testing team had not found during several days of intensive work.

Q: What can information security learn from Moneyball?

Information security can learn that detailed measurement may outperform intuition, established authority, and larger budgets. Baseball teams once made multimillion-dollar player decisions based largely on scouts' judgments. Statistical analysis enabled early adopters with limited resources to choose more effectively, reach the playoffs, and outperform teams whose budgets were sometimes an order of magnitude greater.

Q: Why can data-driven security decisions create fear?

Data-driven security decisions can create fear because credible measurements may expose ineffective investments and poor historical choices. They can also challenge the authority or employment of people whose roles depend on traditional judgment. As security metrics improve, organizations may discover that familiar systems, training efforts, or costly practices have little effect on business security and are therefore degrees of freedom.

Q: How can security teams identify what actually matters?

Security teams can identify what matters by measuring outcomes and testing whether changes in particular controls, systems, or practices alter those outcomes. Variables that change without affecting the result may be degrees of freedom. The process requires moving beyond assumptions and examining evidence, much as the software testers isolated the Sticky Keys interaction after eliminating numerous irrelevant explanations.

Q: When does an industry become more mature in its decision-making?

An industry becomes more mature when it can quickly determine which variables influence outcomes and which do not. That transition replaces superstition and precedent with evidence-based decisions. Baseball advanced through detailed analysis of pitches, swings, and player performance. Information security can undergo a similar transition by developing metrics that reveal which investments materially improve protection and which merely consume resources.

Summary & Key Takeaways

• Information security has long struggled to prove whether purchased systems, training campaigns, and other costly measures actually improve security. Better data and analytics could provide the missing ground truth, allowing organizations to separate consequential variables from irrelevant ones and make decisions based on demonstrated outcomes rather than assumptions, habits, or precedent.

• A software testing story illustrates the difficulty of identifying what matters. After a two-year-old caused a server product to crash, testers investigated numerous irrelevant possibilities before discovering that sitting on the Shift key activated Sticky Keys, which conflicted with a named pipe and triggered the failure.

• Baseball demonstrates how an established field can shift from intuition to measurement. Statistical analysis challenged scouts' traditional authority and created fear that poor historical decisions would be exposed. Early adopters nevertheless gained an advantage, enabling teams with limited budgets to outperform competitors whose budgets were sometimes an order of magnitude larger.