I was recently asked the question, “Where does Security belong in an organization?”

It is an intriguing question, and one that I think about quite often.  Currently most CSOs report to the CIO or CTO.  In a few, rare cases, they report to the Chief Risk Officer (CRO) or Legal.  I hear security professionals expressing the belief that the CSO should report directly to the CEO in order to make their voices heard and make security a priority in the organization.

Where does the CSOs role come from?

Decades ago, IT security consisted of rudimentary tools.  Firewalls were one of the first bastions and “network security” became synonymous with “security”.  As attacks evolved, security broadened, as did the tooling to protect against them.  At the same time breaches also became more public, costly, and embarrassing.  Security professionals began clamoring for organizations to take security more seriously.  There was a strong belief that security risks were lost on managers and executives.

With this perceived disconnect, various organizations attempted to step in and remedy the problem.  Government regulation (Sarbanes-Oxley, banking regulations), industry regulation (PCI, ISO 27001, SOC 2) appeared in an effort to mandate security risk mitigation. 

In the midst of these movements, security professionals began to clamor that they “need a seat at the table” – a reference to needing a direct audience with the CEO and board-of-directors.  While some regulations mandated the creation of “Chief Security Officers” and “Chief Privacy Officers” with accountability, none have (yet) stated a mandatory reporting structure.

It is in this landscape that we pose the question, where should the CSO report in an organization.  I would put forward that where the CSO should report may not seem obvious for some reasons not often considered.

“Risk and Opportunity are two sides of the same coin.”

Security’s role is to help the organization realize the opportunity its pursuing by not falling into traps that can destroy the opportunity.  If you think about how security operates, you might think, at face value, the preceding statement fits with what you think security does today.  However, there is some subtlety in the statement that might be overlooked.  Consider the word opportunity, and how the goal of security is “…to help the organization realize the opportunity…”. How often have you heard a security team say, “…over my dead body…”, “…that will never happen…” or something similar that reflects an outright resistance to a project, a change, or a technology.  I’ve heard it said about Online Internet Banking in early 2000s.  I heard it said about mobile payments in the early 2010s.  My point isn’t that opportunities proceed despite security, but rather that they have proceeded by solving for security.  Each of those technologies had detractors who looked for reasons it shouldn’t work, rather than creating solutions that made it possible for it to work.

You might ask what this has to do with where a CSO and their organization reports.  I’ll give you a very simple answer – the CSO and their security group should exist in the structure the enables them to best collaborate with groups generating opportunities.  Security should be just as embedded in building solutions the lead to opportunities as everyone else in the organization.

It is all about how to contribute

My view of where a CSO and their organization exists has much more to do with how they can collaborate and contribute the most in new initiatives.  In my view, the CSO is there to contribute and collaborate on building success for an opportunity by understanding risk and how to mitigate it.

Based on that view, a CSO should:

• Ensure that new opportunities can resist the most likely threats that can disrupt them by examining and measuring the probabilities of threats and their impact, communicating that information to those building the opportunity, and working collaboratively to devise mitigation for those risks most likely to disrupt the opportunity.  (If you say they must resist all threats, then we need to have a separate discussion on how every executive and manager decides to pursue an opportunity with no absolute certainty that the opportunity will succeed.)
• Ensure that security, regulatory, and compliance regimens can be met by creating solutions that meet the requirements of the regimens in ways that allow the opportunity to proceed.  Help the organization design, build, and operate the opportunities in a way that meets the security, regulatory, and compliance regimens. 
• Focus on early collaboration, early engagement, design, early testing, and early feedback.  When the focus is made earlier in developing the opportunity, efficiency increases, and the flow of work becomes more rapid.

There is a question to all of this – who enforces “the rules” when a team, or opportunity does not follow the security, regulatory, or compliance regimens.  In my opinion, that is up to the executive team, and board of directors.  The CSOs role is to provide guidance, and insight, not to enforce or punish.

The CSO should:

• Measure compliance, identify security incidents and risks that can disrupt the opportunity, and refine designs with a focus on making the opportunity a success, and ensuring it stays that way.

The CSO should not:

• Play enforcer.  This conflates the roles of auditor and of guide.  Enforcement is an after-the-fact activity that too often occurs when evaluating a production environment, or a solution that is well on its way.  Guidance occurs, and effective collaboration occurs when there is fast, early feedback to teams that are building the opportunities.

When security thinks of itself as an audit or enforcement function it separates itself from those creating opportunity.  It creates an “us-vs-them” dynamic that is counter to building.  What is needed is an “Us” approach that helps to create solutions that are secure, that meet regulatory and compliance regimens, and protect the opportunity from risks that can disrupt.

So where should the CSO report?

At the end of the day, where the CSO reports should be a reflection of where she contributes to the opportunities of the organization, rather than where she can wield the largest stick of punishment and enforcement.

I do not mind if the CSO exists outside the IT department, but only if they collaborate and contribute in opportunities and initiatives outside of IT.  If the CSO is closer to a Chief Risk Officer and works with every business unit to identify, measure, and treat their risks – whether its identifying criteria for accepting or disqualifying job applicants, measuring impact of workplace safety, or devising strategies to ensure continuous availability of business operations.  A CRO should report at an executive level, but care should be taken not to simply conflate the role of CSO and CRO.  The role and responsibilities of a CRO within a financial services organization is much broader than the skills of most CSOs.

If the CSO is only focused on IT issues, then that CSO should remain within the IT organization, and report to the CIO.  Their role is to identify and prioritize security risks need to be addressed for the sake of the success of the opportunity and collaborate with the rest of the organization on the design, building, and implementation of mitigations against these risks.

CSOs should not lose sight of their role, or that they are one of many parts to making opportunities in an organization.  Security issues are only one of many risks that can make an opportunity fail!  While breaches can cause losses, delayed projects cause losses in sunk costs and lost opportunities.  This should by no means diminishes the role of the CSO as their work helps an opportunity succeed as much as any other part of the organization.  But let’s not inflate the importance of security over the need for an organization to take risks, experiment, and pursue new opportunities.  For the CSO, that should be an opportunity to help the organization take these chances in ways that balance risk and opportunity.

And if you really want me to give a hard opinion, I do not believe it is necessary for a CSO to report to the CEO. My view is that a CSO is an informer, a designer, and a collaborator. They can communicate to a CEO, but they are not the sole mouthpiece of risk and enforcement.

First, a disclaimer…this post is *not* about bashing or ranting about Equifax’s security practices. Why? Because I do not have first hand knowledge of what they did or did not do, or what specific exploits and vulnerabilities were leveraged throughout the kill-chain of the event. Frankly, it’s likely only the security investigators (internal and external), legal team, and outside counsel will ever know the details. Which is just fine by me. If you wonder why, then you’ve obviously never been involved in a breach and the subsequent investigation. There is a lot of conjecture (some logical, some not so logical), and lot of hand wringing, certainly a lot of drinking (after hours), and a whole lot of lost sleep and hair (if you have any to begin with).

So why would I mention that?

Because I want to rant for a moment about the security community and the press who seem to have taken issue with how Equifax was breached.

This has nothing to do with their response to the breach.  Lets set aside Equifax’s horrible response after the breach. I will not condone, support, or even pretend to empathize with their response. To put it mildly, their response to the breach sucks. You were breached. Mae Cupla, and treat your customers, constituents, not-so-willing-pubic-whose-data-you-have like your own injured child who you just accidentally knocked off a ladder and gave the a lump on the head (and maybe a concussion).

Let’s instead talk about the blame we seem so eager to apportion.  Security professionals, take note of something we always say:

– It is not “IF” you will be breached, but “WHEN”

So suddenly Equifax is evil because they were breached?

You may counter, “but they had a vulnerability that was *3* months old!!!!”

Um, yeah….about that. Let me ask you how old the vulnerabilities are on your laptop that you use for your pen-testing. And if you are a CISO or other security professional employed at a company, and you believe you patch your public facing systems perfectly in less than 90 days, you are *woefully* uniformed, I would argue “naive” in understanding how companies work, and not plugged into something called “risk acceptance”. Ouch, I think I just touched some nerves, but let me assure you, this is not personal. It is about the dynamic of an organization – something that outsizes the best of us.

Again, I cannot say this is Equifax, but I can say that nearly every company I’ve come in touch with struggles with this same problem.

Security Team: “Bad vulnerability, and its out there exposed to the Internet. We must patch it right away!”
Development Team: “Can we test this first? Its probably going to break the application.”
Business Team: “This is a really critical app for our business group. Please don’t break it.”
Managers: “Don’t break the app. Can this wait?”
Executives: “We’re listening to all the people here, and can we please just not break things? Lets take it slow and test.”
Development Team: “We have features to get out that are a priority and are scheduled to go out in the next three weeks for a customer.”
Business Team: “Please don’t interfere with this critical customer need.”
Executives: “Can we please not break things…”
Development Team: “The patch breaks something. It will take us a couple of months to figure out what. Meanwhile we have these other features to get in.”
….

See a trend? I don’t want to represent this as being an endless cycle. Reality is (at least for the organizations I’ve worked with) they do eventually, in a fairly reasonable period of time (which I will admit is a *very* subjective assessment) get around to figuring it out and fixing whatever is broken by the patch. Some organizations are great at it and it might take one or two sprints to figure it out. Others, either other priorities or their backlogs are long, and maintenance work doesn’t fit into such a high priority, but they still get to it within 3-6 months. In some cases, depending upon the complexity of what a patch breaks, that’s pretty darn good. And if you are skeptical of that, you need to spend a bit more time embedded in a development team.

I remember quite a few years ago listening to a talk at BSidesSF (one of the early years) from someone whose day job was to teach companies how to write secure code, and evaluate code for security vulnerabilities.  He talked about a program that a customer asked them to write, and how, in their efforts they found that they committed exactly the same secure programming mistakes they lectured their customers to avoid.  They had vulnerabilities in their code, that were easily exploitable.  They found that deadlines made them take shortcuts and not get around to putting to use all the best practices that they could (or maybe should) have.  And these were some individuals who I had very high regard for in the application security field.  They admitted – “Its hard in the real world to do it right.”

So what should we learn from Equifax?

Security isn’t perfect.  We shouldn’t gang up on an organization just because they had a breach.  Every organization is trying to balance a business opportunity with the risk being posed to those opportunities. Its a balance. Its a risk equation. Its never pretty, but lets face it, most organizations are not in the business purely for the sake of security.  Every control costs money, causes a customer frustration, and has an impact on revenue.  You may say a breach does, and it does, but there is a balance.  Where exactly that balance is can be a subject of great debate because it is not precise, and can never be predicted.

Patching is much more than just “patch and forget”.  Application patching is even more complex.  The alleged vulnerability cited in the Equifax breach was 90 days old.  Even if it was 180 days old, there are factors we cannot even begin to understand.  Competing business interests, a belief that its exploitation couldn’t be leveraged further, a penetration team that didn’t find it or the exposure it could lead to because the applications were too complex to understand, or even human error missing the finding through a reporting snafu.  Stuff happens….no one is perfect, and we shouldn’t throw stones when our own houses have (despite our own protestations otherwise) just as much glass in them.

Ultimately, there are some practices that can help, but I will put a disclaimer here – these may already have been in place at Equifax.  Again, we are human, and systems/organizations are complex.  Complexity is hard to get right.  We also don’t know the full kill-chain in the Equifax scenario.  There may be more things that would help, or for that matter, these things may have been in place, it required even more complex efforts to address the root cause.  That said, here’s some things I would suggest:

• Try to understand every application in your environment and how they tie together.  Knowing the potential chains of connection can help you understand potential kill-chains.
• Create multiple layers of protection – so you can avoid a single failure being the result of catastrophic loss.  You can liken this to the “swiss cheese effect” where the failure occurs at multiple layers (or there aren’t any layers) and the breach easily cascades further and further into systems and data.
• Run red-team exercises with targets as goals (e.g. that big database with customer data, or the AD domain user list).  Let your red team think like an outsider with a fun goal, and flexibility of time to figure out how to get there.  The results will inform you where you can improve primary controls, or where you can add additional layers or protection.
• Patch external systems with far more urgency than internal.  This seems pretty obvious, but sometimes how we represent vulnerabilities is too abstract.  I have found that using the language of FAIR has been an immense help.  Two factors I try to focus on: Exposure (what population of the world is it exposed to) and Skill/Effort to exploit (is it easy or hard).  Given the volume of opportunistic threat attempts (a.k.a. door knob twisting), it makes sense to point to those values as key indicators of what will happen with exposed vulnerabilities.  I once pointed to the inordinate number of queries on a specific service port that a client used as proof that the “Internet knew they were there…” which leads to my last point…
• Communicate in a language that people can understand, and in ways that make it real.  If you talk in CVSS scores, you need to go home.  Sorry, but to quote a favorite line of mine, its “Jet engine times peanut butter equals shiny.” (thank you Alex Hutton, your quote is always immortalized in that fan-boy t-shirt).  Put it in terms like: “The vulnerability is exposed to the Internet, there is nothing blocking or stopping anyone from accessing it, and the tools to exploit it are available in code distributed openly to anyone who has Metasploit (an open-source, freely available toolkit).  The attacker can then execute any command on your server that the attacker wants including getting full, unfettered access to that server, its data, and….”

Those are things I coach my teams on.  Things we should look at and learn from.  Because we need to find data that helps us get better.

One last thing that chafed my hide…

Some people had the audacity to say “…who would hire a CISO with a college major in music…”

Setting aside the rather comical philosophical rant I could make based on UCI’s research on the effects of Mozart on students studying mathematics, I’d like to put forth my own experience.

I hold a Bachelor of Architecture (yes, buildings!) and have a minor in Music, and two years post-bachelors in organizational psychology.  I am a fairly accomplished security consultant (who has done penetration testing and programming) and CISO.  My degree is not a disqualification from being a CISO, any more than a Music degree disqualifies the former CISO for Equifax for having her job.  Simply put, “COMPUTER SCIENCE IS NOT A PREREQUISITE FOR BEING A CISO”.

I have interviewed dozens of CISOs around the world.  Nearly every one of them said they liked having liberal arts majors and people outside of Computer Science fields in their teams because they brought a very different insight and analysis to the team.  It is my opinion that by the time you have reached five (5) years of experience, your college education is largely immaterial.  There are theories and data that college informs you of – such as what a petaflop is, if-then statements, and theory of assymetric encryption, but college does not tell you how to use these skills in the ever-changing dynamic of real life.  I call these skills the ability to analyze, synthesize, and respond.  E.g. the act of design.

For the CISO of Equifax, it is likely that her skills of analytics and design, and her ability to communicate those thoughts to executives were highly skilled.  It is also likely that she had experience with software, with networks, and other technical areas.  I can relate because in my undergraduate education for Architecture we had to take a Pascal programming class in our freshman year.  We had to take a “Computers in Architecture” class.  What I did with it was unique, and I would suspect what the former CISO of Equifax did with her experiences was unique as well.  Putting a blanket assumption over anyone’s experience is ill-informed, and frankly, quite naive.  Have a chat with them.  Know their skills, learn what made them capable and skilled, or at least trusted at what they do.  Then critique what they have brought to the table *today* as a result of all of their experiences (school included, but also all their work since then).

So let everyone put down the pitchforks and stones we were going to throw at someone else’s glass house, and go back to tending our own glass houses and note how someone else’s glass house got broken as a way to learn how to protect our own.  Because what I’m hearing so far isn’t helping, and is based on a lot of arm flapping and people far too interested at pointing at other people’s glass houses than tending their own.

I’d like to tell a story (a mostly real one) that can help you think through how to make your DevOps transition a little smoother, level set some over-exuberance, and ensure everyone feels they are getting a fair shake in a way that is collaborative.

I had a customer whose teams talked endlessly about how they wanted to get to DevOps, continuous integration, and high velocity of deployments.

The challenge is that they talked about DevOps making Deployment going faster.  They wanted rapid deployment, daily changes, and to push code to production every day.  As a result everyone latched onto what they thought it meant.  They talked about faster creation and deployment of new features.  They talked about end outcomes and the excitement of reaching that end goal of daily pushes.  Developers thought they had reached nirvana and could get all the code that was backlogged into production whenever they wanted it. Operations teams thought it meant that development would write cookbooks and test everything and they could focus on undoing technical debt, getting rid of crappy code, and making things work right in production.

Now these are all valid goals of DevOps.  They all are things we want to strive for.  But they were being framed in the legacy biases of Dev vs. Ops.  As an example, someone who is typically production and operations focused could easily quickly admonish the developers for being “unrealistic” in their expectation to jump straight to daily releases and rapid increases in speed and velocity.  You don’t jump straight from typing in code to putting it in production.  At least not in reality, and certainly not with quality.  While there is truth in these statements, any admonishment is going to be perceived by developers as blocker to the speed and velocity they want.  And they’ll push back and say, “Its all Operations being the blocker and slowing us down!”  And they’d be partially right.

So instead of admonishing the developers, we changed our language and effort to focus on one of the source of our issues –  environmental stability. The development and QA environments were unstable, systems were undersized to run any meaningful tests or even run the programs run on production systems, and they did not have representative data to work with.

We started saying “we are going to give you stable Dev and test environments”, “we’re going to increase speed and accuracy of testing”, “we’re going to get you good test data that is as close to current and complete as possible in prod as possible”, “we’re going to give you any data you need to identify, debug, analyze and respond to test and prod failures”.  This shifted the conversation from being adversarial (Devs pointing at Ops and saying their obstructionists) to being collaborative (ooooh, they’re going to give us shiny new toys!).

Ops focused on building a proper development and QA environment that could very accurately depict production.  We first sized resources (hardware, networks) that could support the effort.  This might seem “wasteful” since development doesn’t generate money, why don’t we go with left over systems.  But the point I raised was that development was where the real work was taking place – where undersizing would be a mistake and lead to all the mistakes happening in production, where mistakes cost money.  Lets instead make mistakes in an environment where it doesn’t cost the company money.  This doesn’t mean that we spend exorbitantly, but that we shouldn’t be foolishly cheap.  Development/QA was built in the way that the teams wanted to build production.  It used the tools they wanted to use in production.  And we ignored any further work on production.  Yes, you heard that, we didn’t go after technical debt in production (unless it caused an outage).  Why did we do that?  Because there was no sense in fixing things that we didn’t know yet if those fixes were appropriate.  We needed to test the entire infrastructure, not just the code, as a development effort.  We needed to get code that was tested and optimized and architected the best way through prototyping.  We needed to test building systems, deploying the operating system configured in the way the development teams needed it configured, installing databases, and doing anything else that was needed to give the developers the environment they would expect to deploy on top of.  We needed to do this in an environment where we could make mistakes, learn from them, and correct them – all without impacting generation of revenue.

What we accomplished was a double win.  We gave Developers the resources they needed to be productive.  We gave them tools, stability, data, and capacity to experiment.  We gave them testing tools…and the operations teams got to test right along side of them.  They got to build the tools, build the stability, learn how to handle the data, and build the capacity.  It was no longer pie in the sky but what each Dev team wanted and needed to go faster, and lessons on how Operations could clean up the technical debt in a way that mirrored the Developers’ intention.  It was about how we could positively influence the  lives of our Dev teams, and Ops teams.

The following are quotes or paraphrased notes taken from talks I have seen, podcasts, or general conversations with people I know.  If you feel you didn’t say these words, or wish to correct them, just contact me.

———

Microsoft gets it: you don’t teach programmers to be security people.  You do it for them (or make it hard for them to do it wrong). – Unknown

——–

“Don’t make people security experts, make it easy for people.  Get out of the echo chamber.  Make accessible the message that people care about.  People don’t want to think about security in what they do – they just want it to be there.”  – Josh Corman

——–

“Make things simple and they will do it.  Make it easier so people will use it.”  – Unknown

——–

“People respond to transparency and openness.  When issues are exposed – surfaced.” – Unknown SIRAcon 2016

——–

“We have to accept that its not our risk tolerance that matters as risk practitioners or security professionals.  Its the person accountable for the risk at the end of the day. And until you overcome that your almost a barrier to what you’re trying to achieve.”  – Chris Hayes

——-

“We have to work with the biz to get them to understand the risk, and design with it (for better solutions). This is why security should have 2 parts (maybe 3). A) understand and design ways to mitigate the risk for the new, B) manage risk day to day, operations C) Analyze the performance and effectiveness over time”

———

Risk Manager’s job is helping CSO sell security – sell the project.  Whether its a great big investment decision, or small item – what are the attributes, the Risk and Opportunity measures (estimates and forces at play).  – Alex Hutton

———

Risk Management / Security Metrics is a Security Optimization Program

I recently had a pair of encounters that have opened my eyes further to both the causes of our current messy state of IT affairs, and given me hope for a better future.  In both cases the issue that came up with access to production environments.

In one particular case a user had their access removed – ostensibly on the grounds that their access violated “segregation of duties between development and production”.  There are numerous control frameworks that demand a segregation of production and development environments.  There are even others that say personnel should be fully segregated.  Lets look at where this came from, and what the outcome has been:

• Segregation of duties came about as a control for preventing one person from performing an end-to-end activity by introducing a check that the activity was appropriate.  It started largely as a financial control.  The most obvious is preventing an Accounts Payable clerk from inputting a purchase or payment request, and then processing that payment request themselves – all for the benefit of their own personal bank account.
• This control was extended to IT – especially during the Sarbanes-Oxley days – as a way to ensure that a developer could not introduce ways into the programs to siphon off pennies all for the benefit of their own personal bank account.
• This control was then extended further to include personnel access to anything in production because (again ostensibly) it was believed that sharing information about production would create knowledge that developers could exploit.

Lets be clear.  Controls that prevent the theft of money (fraud) are important.  However the lengths to which this control has been extended has become ludicrous.  What it has done is damage the workflow, trust, collaboration and functioning of the IT department and its ability to support the business needs of all other parts of the company.  How you ask?

• The segregation-of-duties controls are extended to deny developers visibility into the environment, which means their situational awareness of how their programs are running is removed.
• They lose the belief that other groups trust them since their visibility is removed.  They pull up a wall.
• They now view the operation of a program as “someone else’s problem since they don’t let us in”.  The pull that wall up higher.
• They now throw programs over the wall – because “we’re not responsible for them in operations”.  Operations hates when this happens.
• Myriads of other controls flow in to stop-gap the problems that development teams don’t have the visibility to understand.  Testing requirements increase to address the problems since it is believed the problem is in insufficient testing.  The testing becomes cumbersome, laborious, and yet largely ignorant to the problems that happen in production.
• Costs go up, blame goes up, and failures happen…and the speed of work goes down.

Sounding familiar yet?

So how does this fit into my realization?  Access into production for developers is not a bad thing.  Developers should have visibility into application and system logs so they can view the reaction of their code in real world situations.  Developers should have the ability to see elements that are not sensitive.  They likely shouldn’t see sensitive data like payment cards, or encryption keys, but they should be able to see configuration files, data types and definitions.  Give developers what they need to create a feedback loop that is clear, unobstructed, but doesn’t violate regulations.

That being said, developers promoting code into production without checks and balances is a bad thing.  That I think we can agree on, but how does that fly with a DevOps mentality?  How about:

• Changes can go into production once they go through an automated test suite.  They are only available for check-out when they meet that criteria of that automated test suite.
• Production personnel (ops) can promote into production anything that has gone through the test suite and is available for check-out into production.
• Development personnel can check problems and push fixes through this same chain.

If you notice, in the better world, developers have access to view, and monitor the production environment – they have a feedback loop.  In the better world, developers still have to have their programs vetted by a testing procedure before changes are pushed to production.  The key control objective is still met – reduce the probability for fraud – but with controls that keep the collaboration, accountability, and teamwork in place.

Now in the two cases that I came across, both arrived at the same conclusion.  Both believed that visibility was important.  Both believed that it could be achieved.  The challenge was to educate those who have accepted the de facto standard of full segregation without understanding the original goal, and the impact of such a decision.