
IT Security Metrics: Four Pitfalls To Avoid


Tim Ramsay, CISO & AVP, University of Miami
How does a Chief Information Security Officer demonstrate effectiveness? When the opportunity arises, in the elevator or in the boardroom, every CISO must be prepared to answer the question: “How secure are we?” Sometimes this question is driven by a reaction to data breach headlines or angst about the return on investment for prior security expenditures. Whatever the motive, the question is inevitable. CISOs who invest the time to create clear performance metrics, before they are asked, enhance the credibility of their office and build confidence in the security program generally. Even so, information security organizations have long struggled with the details of creating a metrics program. The discipline is still under development, a common vocabulary understood by all is elusive, data collection capabilities vary widely between organizations and few best practices exist. Still, the potential impact of using data to measure the effectiveness of a security program cannot be under estimated. To create meaningful metrics, there are four pitfalls to consciously avoid.
Confusing measurements with metrics
Peter F. Drucker, one of the most prominent management consultants of the twentieth century observed, “What's measured improves”. However, he also stated, “There is nothing so useless as doing efficiently that which should not be done at all.” In other words, it is possible to manage the wrong thing, draw attention to it, even observe statistical “improvement” and reach incorrect conclusions. The key for security professionals is to clearly link activities to outcomes. If there is no relational understanding between an activity and an outcome, any observed change is entirely random. To qualify as a metric, a measurement must be based on a deep understanding of process and connect with an objective management intended. A few, well-defined, agreed upon metrics tied to goals management cares about will provide more assurance than a vast array of sophisticated data extracts from security appliances without context.
Confusing ownership with execution
Owning a metric does not require ownership of all the underlying tasks. Security professionals need to “own”the right metrics regardless of whether or not they are operationally responsible for all the associated activities. For example, even if the CISO is not responsible for all facets of the Identity Access Management (IAM) function, he/she is accountable for the overall effectiveness of the IAM program. IAM represents the keys to the kingdom at every organization. If a CISO only creates metrics for processes he/she has end-to-end responsibility for managing, the enterprise will be exposed. In mature organizations, there’s an understanding security is everyone’s responsibility. Even then, the CISO remains uniquely positioned to measure the effectiveness of the security program, regardless of how functional activities are spread. CISOs must have an institutional view that extends beyond the boundaries of their direct staff. This “ownership” responsibility applies to all elements of the IT asset portfolio including the network, data centers, endpoints, applications and appropriate third party vendors. The less operational responsibility a CISO has for IT functions, the more objective he/she can be in measuring performance because there is no real or perceived conflict of interest. For example, if a security organization is responsible for managing the endpoint security program (antivirus, encryption, etc.), who assesses the effectiveness of the program? Should the security team grade their own homework? As a practical matter, depending on the size of the organization, this may be reasonable, especially if Internal Audit or an external third party periodically assesses the program. Yet, where possible, for the purposes of independence and objectivity, the overall IT structure should attempt to separate strategic functions from operational functions.
Presenting the same metrics to all audiences
The audience and the context matter greatly when reporting metrics. Senior executives ask different questions about security than IT managers. Likewise, end users have different security concerns than line supervisors. Predictably, C-level executives will almost always be concerned about the overall program, risk posture, process maturity, return on investment, and comparisons to peers, the past, standards and goal. Unfortunately, instead of having direct dialogue with various stakeholders about how they would like to measure the security program, some CISOs primarily turn to professional security organizations, consultants and peers. Even if they receive competent guidance, these CISOs miss a huge opportunity to engage their constituents in a valuable conversation that demonstrates good faith. Even when the CISO thinks the metrics are obvious, there is value in listening to stakeholders. For example, CISOs who allow business leaders to jointly create the data classification framework often find they have created true business owners without intending to do so. This, in turn, allows more granular risk-based metrics to be reported.
See Also: Top Inventory Management Technology Companies
Ignoring the dynamic nature of metrics
Just as business models evolve, so must security metrics. New threats are constantly emerging and new tools become available that require security professionals to reevaluate their data collection, analysis and reporting methodology. In addition, changes in business direction or the security maturity of the enterprise may require a new, elevated target for success or an adoption of an entirely different metric. For example, what is the benefit of measuring endpoint encryption on laptops for an organization that has completely adopted both a BYOD (Bring Your Own Device) program coupled with VDI (virtual desktop infrastructure) and there is literally no corporate data on any user computer? In this scenario, the CISO has protected the organization by transforming the computer into a typewriter with no sensitive data. Security professionals have a unique vantage point within an organization to observe and measure risk from a process perspective, end-to-end. Those CISOs who focus on the underlying processes and not just data collection, who take ownership for measuring security outcomes regardless of how tasks are distributed, who adapt to each audience and sallow the metrics program to evolve over time will successfully steer management’s risk posture with objective data and allow everyone in the organization to understand their connection to the security program.
Check This Out: Top Inventory Management Solution Companies
CIO Review Clients : Flagship , PCMI
Media Partner : CIO Review | B2B Online 2020
CIO Review Press Releases : CIO Review | One Stop Systems
ON THE DECK
Featured Vendors
THETA432: Performance, Precision, Efficiency, Visibility - The Key to Incident Response and Answer to the Talent Shortage
EveryCloud Technologies: Delivering Powerful Email Filtering Services" title="Graham O'Reilly, CEO & Co-Founder" style="float:left; margin-right:10px; margin-bottom:20px;" width="60px" height="50px">
EveryCloud Technologies: Delivering Powerful Email Filtering Services
Onepath: A Responsive Info-Security Management Framework – the easier way to dramatically improve your overall info-security posture
Covenant Security Solutions, Inc.: Revolutionary Solutions to Mitigate Security and Compliance Risks
EDITOR'S PICK
Essential Technology Elements Necessary To Enable...
By Leni Kaufman, VP & CIO, Newport News Shipbuilding
Comparative Data Among Physician Peers
By George Evans, CIO, Singing River Health System
Monitoring Technologies Without Human Intervention
By John Kamin, EVP and CIO, Old National Bancorp
Unlocking the Value of Connected Cars
By Elliot Garbus, VP-IoT Solutions Group & GM-Automotive...
Digital Innovation Giving Rise to New Capabilities
By Gregory Morrison, SVP & CIO, Cox Enterprises
Staying Connected to Organizational Priorities is Vital...
By Alberto Ruocco, CIO, American Electric Power
Comprehensible Distribution of Training and Information...
By Sam Lamonica, CIO & VP Information Systems, Rosendin...
The Current Focus is On Comprehensive Solutions
By Sergey Cherkasov, CIO, PhosAgro
Big Data Analytics and Its Impact on the Supply Chain
By Pascal Becotte, MD-Global Supply Chain Practice for the...
Technology's Impact on Field Services
By Stephen Caulfield, Executive Director, Global Field...
Carmax, the Automobile Business with IT at the Core
By Shamim Mohammad, SVP & CIO, CarMax
The CIO's role in rethinking the scope of EPM for...
By Ronald Seymore, Managing Director, Enterprise Performance...
Driving Insurance Agent Productivity with Mobile and Big...
By Brad Bodell, SVP and CIO, CNO Financial Group, Inc.
Transformative Impact On The IT Landscape
By Jim Whitehurst, CEO, Red Hat
Get Ready for an IT Renaissance: Brought to You by Big...
By Clark Golestani, EVP and CIO, Merck
Four Initiatives Driving ECM Innovation
By Scott Craig, Vice President of Product Marketing, Lexmark...
Technology to Leverage and Enable
By Dave Kipe, SVP, Global Operations, Scholastic Inc.
By Meerah Rajavel, CIO, Forcepoint
AI is the New UI-AI + UX + DesignOps
By Amit Bahree, Executive, Global Technology and Innovation,...
Evolving Role of the CIO - Enabling Business Execution...
By Greg Tacchetti, CIO, State Auto Insurance
Read Also
Hybrid Work Has Forever Changed The Need For It In Ramboll
How T-Mobile brought an Un-carrier approach to tech hiring
Every Changing Labor Force
Great Expectations: Balancing the diverse needs of a city in a...
Community Banks And Digital Banking
"Discovery and Delivery" - An Approach to IT Workload Balance
