Why Cybersecurity Progress Is So Hard to Measure

TL;DR

Cybersecurity is not clearly improving because the industry lacks reliable criteria for measuring progress and often prioritizes audits over actual attackers. Organizations should judge security investments by their effects on cost, complexity, risk, and the replaceability of protected assets, then redirect effort from easily replaced data toward intellectual property and other assets whose loss has lasting consequences.

Transcript

All right. Good morning. I have a, a bit of a head cold, but I will do my best to suffer on. I think, uh, Philip gave it to me, but... Slides, please. Okay. So the question, are we getting better, why we don't know, and what we can do about it. So I really don't care about my backstory per se, um, but just to give you some perspective, I'm currentl... Read More

Key Insights

• Cybersecurity progress is difficult to prove because the industry has very poor indicators of success. More than a breach a day and a growing range of adversaries show worsening conditions, but these observations alone do not provide an agreed standard for measuring improvement.
• Security conditions change through five major forces: evolving threats, compliance obligations, technology shifts, business imperatives, and socioeconomic pressures. A change in any one of these areas can alter an organization's total cost, operational complexity, and risk, making a single progress metric inadequate.
• Compliance can become a substitute for confronting actual threats because an organization might be hacked but will be fined if it fails an audit. This certainty drives some companies to spend only on passing audits, even when that focus does not address their real attackers.
• Technology change expands both the scope and complexity of security work. Virtualization, cloud computing, mobility, and bring-your-own-device practices require different controls, while SaaS, PaaS, and IaaS applications cannot necessarily be managed like systems inside a traditional organizational environment.
• Business and socioeconomic pressures directly shape security decisions. Leadership priorities, supply-chain collaboration, disaster recovery, budget cycles, economic downturns, outsourcing, and premature cloud adoption can all change exposure, while cultural and economic conditions can also contribute to new forms of activist threats.
• Security strategy should begin with a clear reason for protecting systems and information. Organizations commonly explain what they do and sometimes how they do it, but rarely articulate why, even though shared values and mission can guide priorities and make subsequent decisions follow more naturally.
• Asset replaceability is a useful measure of security consequence. Credit cards are highly replaceable and usually cause limited inconvenience after compromise, while trade secrets and intellectual property may sustain the business, drive profit, and create lasting damage when they are lost.
• Security spending is misaligned when approximately ninety-five percent of effort targets credit cards while less replaceable assets receive little or no protection. The speaker argues that falling criminal interest in card data further strengthens the case for redirecting resources toward consequential intellectual property loss.

Questions & Answers

Q: How can organizations measure whether cybersecurity is improving?

Organizations can evaluate progress by defining explicit criteria across evolving threats, compliance duties, technology changes, business imperatives, and socioeconomic conditions. Each category should be assessed for its effect on total cost, complexity, and risk. Breach counts or audit completion alone are insufficient because they measure visible events or activity without showing whether the organization is reducing consequential harm.

Q: Why do cybersecurity programs focus too heavily on compliance?

Compliance receives disproportionate attention because audit failure and associated fines are predictable, while a future attack may seem uncertain. The speaker describes Fortune 100 companies that spend nothing beyond passing their audits. This approach turns the auditor into the threat being managed and can divert attention from real attackers, changing technologies, and assets that matter more to the business.

Q: What factors make cybersecurity risk continually change?

Cybersecurity risk changes through five interacting forces: new adversaries and attack methods, expanding compliance requirements, disruptive technology, shifting business priorities, and socioeconomic pressures. Examples in the talk include nation-states, activists, privacy laws, virtualization, cloud computing, mobility, supply-chain collaboration, outsourcing, budget cycles, and economic conditions. Each force can increase cost, complexity, or organizational risk.

Q: Why is asset replaceability important in cybersecurity?

Asset replaceability helps distinguish inconvenient losses from damage with enduring consequences. A compromised credit card can generally be replaced, and the speaker says the maximum fifty-dollar U.S. charge is usually waived. Trade secrets and intellectual property are less replaceable because they help keep a company operating and drive profit. Protection priorities should reflect that difference in consequence.

Q: Why should companies protect intellectual property more carefully?

Intellectual property can sustain a company's business and profitability, yet the speaker argues that many organizations spend nothing protecting it while concentrating on payment-card compliance. He tracked admissions from eighty-six Fortune 100 companies that had lost intellectual property during a 12-month period. Such losses can be far less reversible than replacing payment credentials after a breach.

Q: How does new technology increase cybersecurity complexity?

New technology changes the environment that security teams must protect and expands the scope of their responsibilities. The talk identifies virtualization, cloud computing, mobility, and bring-your-own-device practices as successive disruptions. It also notes that SaaS, PaaS, and IaaS applications cannot be managed in exactly the same way as systems in a traditional internal environment.

Q: Why are breach counts not enough to judge security progress?

Breach counts show that incidents occur, but they do not establish a complete criterion for improvement or decline. The speaker notes more than a breach a day, more adversaries, and more types of adversaries, while also observing that organizations generally have not gone out of business. Progress requires measures connected to consequences, costs, complexity, risk, and protected assets.

Q: How should security leaders choose what to protect first?

Security leaders should start by clarifying why their security program exists, then prioritize assets according to the consequences and replaceability of their loss. Easily replaced payment data should not automatically consume most resources while trade secrets receive none. Decisions should also account for actual adversaries, technology changes, compliance demands, business priorities, and socioeconomic constraints affecting the organization.

Summary & Key Takeaways

• Cybersecurity progress cannot be established merely by counting breaches, adversaries, or compliance activities. The industry lacks agreed criteria for deciding whether conditions are improving, worsening, or remaining stable. Meaningful evaluation must consider how evolving threats, regulation, technology, business priorities, and socioeconomic pressures change an organization's total cost, complexity, and risk.

• Compliance can displace threat-focused security because audits and fines are predictable, while attacks appear uncertain. Some organizations spend nothing beyond what is needed to pass an audit, even as changing technology and multiplying regulations expand their exposure. Security programs should therefore distinguish visible activity from measurable effects and address weaknesses across several categories of change.

• Asset replaceability offers a consequence-based way to prioritize protection. Credit cards are replaceable and may cause limited inconvenience, yet they receive most security attention. Trade secrets and intellectual property can sustain a business and drive profit, but may remain neglected. Security resources should move toward assets whose compromise creates enduring, consequential harm.