IT Security Metrics: Four Pitfalls To Avoid

Tim Ramsay, CISO & AVP, University of Miami

1139

1997

389

Tim Ramsay, CISO & AVP, University of Miami

How does a Chief Information Security Officer demonstrate effectiveness? When the opportunity arises, in the elevator or in the boardroom, every CISO must be prepared to answer the question: “How secure are we?” Sometimes this question is driven by a reaction to data breach headlines or angst about the return on investment for prior security expenditures. Whatever the motive, the question is inevitable. CISOs who invest the time to create clear performance metrics, before they are asked, enhance the credibility of their office and build confidence in the security program generally. Even so, information security organizations have long struggled with the details of creating a metrics program. The discipline is still under development, a common vocabulary understood by all is elusive, data collection capabilities vary widely between organizations and few best practices exist. Still, the potential impact of using data to measure the effectiveness of a security program cannot be under estimated. To create meaningful metrics, there are four pitfalls to consciously avoid.

Confusing measurements with metrics

Peter F. Drucker, one of the most prominent management consultants of the twentieth century observed, “What's measured improves”. However, he also stated, “There is nothing so useless as doing efficiently that which should not be done at all.” In other words, it is possible to manage the wrong thing, draw attention to it, even observe statistical “improvement” and reach incorrect conclusions. The key for security professionals is to clearly link activities to outcomes. If there is no relational understanding between an activity and an outcome, any observed change is entirely random. To qualify as a metric, a measurement must be based on a deep understanding of process and connect with an objective management intended. A few, well-defined, agreed upon metrics tied to goals management cares about will provide more assurance than a vast array of sophisticated data extracts from security appliances without context.

Confusing ownership with execution

Owning a metric does not require ownership of all the underlying tasks. Security professionals need to “own”the right metrics regardless of whether or not they are operationally responsible for all the associated activities. For example, even if the CISO is not responsible for all facets of the Identity Access Management (IAM) function, he/she is accountable for the overall effectiveness of the IAM program. IAM represents the keys to the kingdom at every organization. If a CISO only creates metrics for processes he/she has end-to-end responsibility for managing, the enterprise will be exposed. In mature organizations, there’s an understanding security is everyone’s responsibility. Even then, the CISO remains uniquely positioned to measure the effectiveness of the security program, regardless of how functional activities are spread. CISOs must have an institutional view that extends beyond the boundaries of their direct staff. This “ownership” responsibility applies to all elements of the IT asset portfolio including the network, data centers, endpoints, applications and appropriate third party vendors. The less operational responsibility a CISO has for IT functions, the more objective he/she can be in measuring performance because there is no real or perceived conflict of interest. For example, if a security organization is responsible for managing the endpoint security program (antivirus, encryption, etc.), who assesses the effectiveness of the program? Should the security team grade their own homework? As a practical matter, depending on the size of the organization, this may be reasonable, especially if Internal Audit or an external third party periodically assesses the program. Yet, where possible, for the purposes of independence and objectivity, the overall IT structure should attempt to separate strategic functions from operational functions.

Presenting the same metrics to all audiences

The audience and the context matter greatly when reporting metrics. Senior executives ask different questions about security than IT managers. Likewise, end users have different security concerns than line supervisors. Predictably, C-level executives will almost always be concerned about the overall program, risk posture, process maturity, return on investment, and comparisons to peers, the past, standards and goal. Unfortunately, instead of having direct dialogue with various stakeholders about how they would like to measure the security program, some CISOs primarily turn to professional security organizations, consultants and peers. Even if they receive competent guidance, these CISOs miss a huge opportunity to engage their constituents in a valuable conversation that demonstrates good faith. Even when the CISO thinks the metrics are obvious, there is value in listening to stakeholders. For example, CISOs who allow business leaders to jointly create the data classification framework often find they have created true business owners without intending to do so. This, in turn, allows more granular risk-based metrics to be reported.

Ignoring the dynamic nature of metrics

Just as business models evolve, so must security metrics. New threats are constantly emerging and new tools become available that require security professionals to reevaluate their data collection, analysis and reporting methodology. In addition, changes in business direction or the security maturity of the enterprise may require a new, elevated target for success or an adoption of an entirely different metric. For example, what is the benefit of measuring endpoint encryption on laptops for an organization that has completely adopted both a BYOD (Bring Your Own Device) program coupled with VDI (virtual desktop infrastructure) and there is literally no corporate data on any user computer? In this scenario, the CISO has protected the organization by transforming the computer into a typewriter with no sensitive data. Security professionals have a unique vantage point within an organization to observe and measure risk from a process perspective, end-to-end. Those CISOs who focus on the underlying processes and not just data collection, who take ownership for measuring security outcomes regardless of how tasks are distributed, who adapt to each audience and sallow the metrics program to evolve over time will successfully steer management’s risk posture with objective data and allow everyone in the organization to understand their connection to the security program.