The Three Phases to DevOps in Security

Many of those who aspire to create a high-performing security function within a company are looking at DevSecOps and what it represents.  This is laudable, as the concepts that are represented in DevSecOps mirror many of the successful organizations I’ve experienced, as well as the views of dozens of CSOs that I’ve interviewed since 2010.  The CSOs I interviewed often reflected that many of the skills that they valued were not traditional technology skills, but instead skills in critical thinking and collaborative discourse. (Before you toss aside this assertion, bear in mind, they also asserted that technology skill is also needed, but not in isolation)

When a group of us were reviewing very early drafts of “The DevOps Cookbook”, many of us felt something was missing in its approach.  It was David Mortman who first put to paper what many felt was an underpinning concept or theme that was needed: culture.  DevOps depended upon a culture – one that was seemingly at odds with how things were currently being done, and that required buy-in to change.  It required an agent of change, and long-term commitment to overcome disfunction within an organization that may feel counter to existing dogma.

The journey I’ve taken over the past eight years has allowed me to codify some of the successful approaches I’ve taken and understand the why around their success.  This is my collection of ideas, and the basis for them.  It maps the path I’ve charted with my teams towards a culture of broad collaboration, empathy for the “customer”, and a willingness to take chances and learn.  The results I and others have seen are quite rewarding, and you’ll see how each played out in my stories.  These are far from the end-all of approaches, but if they help give you a jumpstart in your journey, then this has achieved what I hoped.

People, Process, Technology

We all know the mantra (principle) of – People, Process, Technology.  It is a fantastic model to explain how things should be.  I even stack them much like I might order Maslow’s Hierarchy of Needs.  People are needed to operate an environment and are the culture of doing and knowledge.  These people build processes that do things that reflect their views on how things get done.  And then you build technology to facilitate the speed of those processes that are designed by these people.

That is wonderful if you are an anthropologist examining how an organization is.  The challenge is that this is rarely how people try to transform their organizations.  They (mistakenly) start upside down.

Technology, Process, People

How many of you have watched an organization declare it’s starting its “DevOps Transformation” and bring in a bunch of technology tools (automation, deployment, cloud)?  This is the “technology shall cure all our ills” club.  I will often tell them, if a process is broken and bad, then all technology will do is make the “bad” faster.  If your process for approving user access requires five different approvals from people who have no idea what they are approving, what system it refers to, or what data it exposes, all technology will do is make that inappropriate access happen faster.  Garbage In, Garbage Out, at Speed.  Have you really made anything better?  How would you know?

How many of you have seen an organization build an isolated team in IT and give it the title “DevOps”?  This one irritates me – if you call your team DevOps, you don’t get it.  Either this implies that only this one team needs to do DevOps, or more likely a naïve notion that it’s all about the automation.  Have you helped the company move forward and improve? How are other groups improving and work across the organization getting better?

Security teams are no better at fostering DevOps.  Too frequently I encounter teams sitting behind walls throwing darts (findings) over those walls at groups they barely know.  This grates me more than someone calling their team “DevOps”.  I call this the “We Do This, You Do That” club.  (By the way, I also see Development teams and Infrastructure teams doing the same).  How do your findings relate to what the company is trying to achieve?  How do they relate to the company’s tolerance for risk?

DevOps is the Journey

You would think that by now people would have learned what DevOps is, but instead DevOps has been miscast as purely automation, or more commonly, deployment tooling.  Let’s get over this myth.  Tooling is an outcome.  Even refinement of work and processes is an outcome.  Make no mistake, I love the technology solutions that have come out of the DevOps movement – methods and tools that have refined the flow of work and that have increased speed of getting things done.  But these solutions are the outcome.  DevOps is the ongoing journey of getting there.  It’s about how we work together with a common goal of making things better (maybe even faster and stronger…) in a way that makes it possible to focus on the real customers, (blamelessly) identify inefficiencies, collectively learn and make leaps of faith, and create rapid and large shifts in how we do things.  It’s a mindset, or I would say, a culture change that allows us to get to that state where we can make these changes.

My Strategy of Change

When I start working with an organization, I put most of my effort into organizational behavior.  The words I use for this are: Embedded, Collaboration, Discourse, Learning, Growth and Refinement.  I’ll concede that there is often badly broken technology or nasty compliance failures – but even these situations I use as an opportunity to teach Security, Development, Infrastructure and Operations teams to work together and learn the cultural workings of DevSecOps.

I first focus on changing how people work and think – their perceptions, understanding, as well as interpersonal and organizational interactions.  I call this Changing People.

I next weave in changing the processes of working – how they communicate, problem solve, and learn.  This overlaps with changing people and can even overlap with changing the company’s operational processes as people try to refine how they work.  I call this Changing Process.

Lastly we look to evolve the processes, tools, and technology used in our operational work.  This can and usually does include changing security controls and operational processes as well as looking at techniques of refinement.  I call this Changing Technology – but in reality it’s about everything that DevSecOps can consume, refine, and make better.  By this time the ideas will start flowing, and the DevSecOps machine is in motion.

Change People

My objectives are to lead the team towards collaboration, communication, discourse, and learning while avoiding being anonymous, disconnection, and debilitating blame:

• Building “emotional capital” with customers
• Broad collaboration as the core to succeeding
• Making the Team feel valued – contributing
• Using Empathy & Discourse to collaborate and solve problems
• Leading by Example

1. Meet Everyone (the Customers): The very first thing I do with a Security team is ask them, who do you know in the organization.  So far, other than one lone individual at one company, they respond just with other IT people – usually support desk, or infrastructure (usually networking).  My response is to task them with getting out and meeting all of the company.  I remind them that what the company does pays their salaries and bonuses, so it would be really good if they knew what that was.  We set up a grand tour where we meet every business unit within the company, and the Security team is tasked with only one task – listening.  They visit Marketing, Sales, Manufacturing, Distribution, Finance, Legal…any and all groups. I challenge the Security team to ask: “What is it that your department does?”, “How does what you do provide value to the company?”, “What keeps you up at night?”, “If you’re department wasn’t available, what would happen?”, “What processes are critical?  What technology is critical for you?”

Oh, and I remind them that there isn’t a wall between IT and “The Business”.  IT is part of the Business (thank you to David Schenk for that mantra).  Further references to “The Business” as a “them” costs them a quarter in a cookie jar.

Result: The team finds out who their customer is.  They will gain an appreciation for what the company does, and what is important to it. Their customer’s value, concerns, and problems become real.  Now the things that Security thinks about can become grounded in what the organization values.  There will be an affinity and empathy towards what the organization does, what pain points exist, and how Security and its actions have an impact.  It Changes the team’s approach from an abstract “Do this so the company doesn’t fail!” to “This will help distribution because they system won’t be disrupted!” or “Charge records will get to Fraud Prevention on time.”

2. Communication: I mandate communication patterns between the teams.  I set down a few rules, many of which will sound like some training you had at some HR event:
   • If your email exchange goes beyond two messages, make it a phone call.
   • Better yet, always start with a phone call. Email is only for transmitting data (files, file manager…)
   • No, Better yet, if you can, walk over and talk to the person face-to-face (I do it by the theory “Managing by Walking Around”.
   • Group meetings are either Face-to-Face or with Video. Video is good when everyone is remote (global).
   • Communicate frequently – have team group meetings and everyone attends. I have one-on-one’s weekly to ensure people feel listened to.

Result: The team knows each other, their faces, and who each other is.  The team learns that most of communication is about facial expressions, vocal tones and things that don’t transmit via email.  Do not let people become anonymous.  Encourage people to feel included.

3. Collaboration & Discourse: During meetings encourage feedback and contribution from all team members in priorities, learning, teaching, and what gets done. I have found that putting people on the spot for feedback doesn’t work well for those that may be more introspective, however making it clear that feedback is welcome, and will be considered opens the opportunity for them to speak.  This is achieved by making it clear that you expect your ideas to be challenged, and that you allow the team to do so.  Consider any feedback you do receive carefully – testing it with the team members offering it, examining how it can disprove your ideas (not how it confirm them).  Make sure the comments are focused on the idea, not the person suggesting it.  We all have ideas that have faults, so no sense blaming.  Rather it is better to refine the idea which becomes a learning experience.  You show value in their ideas and feedback by publicly considering it.  As a manager, allow your statements to be challenged.  Ask your team to disprove them – how could my idea be disproved?

Another technique I use is to ask each person in the team to come with updates on what they are doing so that they understand that everything going on is important and we should discuss it together.  Give them praise publicly for presenting the idea (not just when its right).

Result: You’ll be modeling what you expect your team to do in their interactions.  You’ll surface assumptions, find faults in designs and ideas, and gain a lot of opportunities to teach, and learn!  You’ll create feedback loops – a willingness to discuss openly any issues, problems or concerns.  You will do it in a manner that is open, lacking in blaming the individual, but focusing on the idea.  You will create an environment where people will feel they can participate.

4. Be Willing to Fail: Model this from the top. Admit when you make mistakes.  Give others in the team credit and make it wildly public.  Recognize success globally but keep mistakes internal.  If a team member makes a mistake, take the blame on your shoulders to address, and have the conversation one-on-one with the team member.  Understand the issue, and encourage the learning process.  As the organization as a whole learns blameless environments, you can let mistakes be examined more broadly, but until that adoption occurs, you need to ensure that the team knows that you won’t hold mistakes against them (unless they are systemic and chronic).

Result: You’ll have a dedicated, loyal team.  One that sees learning as a sign of strength.  One that feels they contribute, they’re recognized, and that faults, while always painful and frustrating, will be less so – that they feel they can move forward, learn, grow and correct what goes wrong.

Change Process

1. Allocate Expertise: I take stock of the team, their expertise, strengths, and of course challenges. I also ask what their interests and goals are – what do they like doing, what do they want to do?  With this information I divvy up responsibilities across the Security team.  While the structure depends on needs of the organization, and available skills, I make sure I’ve created comprehensive coverage of responsibilities.  I then collectively let the team know what my thoughts are, let them challenge them, point out what I might have missed, what things need to be added, and where someone feels strengths are not being leveraged properly.  Ultimately its about recognizing expertise in the team, and making sure that expertise is externalized – made public so that everyone knows who they can turn to.

Result: Recognition of expertise in your team, and point to them as the go-to for answers

2. Embed in Projects: Now to break down more walls.  This is how I ensure that the team not only learns about what is going on in the organization, but also participates in creating the solution.  I assign the more experienced security people in my teams (those with broad insight) to projects within the company.  If the effort has a significant need for security, they become the Security Program Manager – the person who triages all requests from security, and who acts as liaison between the project personnel and security specialists within the security team.  This Program Manager needs to be very involved – participating in as many project meetings as possible, engaging with the project personnel, regularly communicating needs, and “Managing by Walking Around”.

I’ve made this arrangement at every client I’ve led.  I’ve had some people take to this like a fish in water – they love the interaction and actively participate with the project team, feel part of the team, and take its success personally.  In one particular case we attended a new project initiation.  We listened, recognized that security was not a material consideration for the project, provided non-security feedback and questions, and were rewarded with a big thank you.  They appreciated our input and made a habit of inviting us for every new project they considered.

Result: Engaging and Embedding in projects.  Participating and having ownership for the participation.  Create low friction, high return work environments where Security is perceived as being invested in the success of the project – through the time committed and the willingness to listen and care about the goal of the project.

3. For Every Control You Implement You Must Give Something Back: This one probably sounds like process, but at its heart is empathy. Security teams have a tendency to impose controls that make tasks harder or take longer.  This is a problem for those people in the company trying to get their work done in time for a deadline imposed by their manager.  In an effort to meet the deadline they will be willing to take any steps to achieve that goal – including side-stepping security.  Security needs to empathize with this.  Hence my rule.

I ask my security teams to look at what their requirements are taking away in terms of efficiency.  How much time is lost because people have to go through the security control (perhaps versus the old way they did things, or versus the intended design).  I’ve had discussions about complex passwords being a requirement, and I ask about the challenge of remembering complex passwords, versus something less cumbersome to remember – like a biometric attribute (a finger) that is rather hard to forget.  Or multi-factor authentication where a token is something to lose, but using a mobile device is less likely to be forgotten because it is ingrained in our culture and daily lives.  Even elements such as change control, or segregation of duties can be examined carefully to see what is really the objective behind the controls, and how can it be arrived at in a manner that is far less cumbersome and obtrusive, yet deliver the same level of resistance to a threat.

Result: A mentality around the potential effects of security, and a thoughtful approach that looks to minimize that impact.  A view that Security Team actually cares and is sensitive to personal success.

4. Prioritize – Be Great at Important Things: This is where I insert a bit of Security – but where understanding the customer comes strongly into play. Nearly every organization struggles with how to prioritize its work.  For Security Teams, this includes the endless list of “must have” projects, tickets, vulnerabilities, and audits.  How do you decide what goes first?  Some use a thumb in the wind, while others claim “professional expertise”.  I prefer measures and collaboration in the team.  I force the team through what I call “Risk Week”.  It’s a week-long session (that gets shorter over time as they get great at it) where we create our Risk model and mitigation priorities for the year.  It is a highly collaborative effort.  It includes revisiting all the organizational groups.  It includes assigned responsibilities within the team so they all participate.  It involves learning how to measure and make the exercise objective and repeatable.  It involves presenting their ideas, each participant challenging assumptions, and creating active discourse as priorities are weighed.  We even include an executive presentation where the team is welcome to present so that they gain the experience and the exposure.

Result: Risk Assessment that is based on the company’s goals and priorities, as well as reinforcing the collaborative nature and interaction that we want to foster.

5. Manage to the Priorities:  Everyone has had the situation where a problem or finding crops up, and suddenly there is belief it needs to be the foremost problem we solve.  It is a “hair-on-fire” moment, and the belief is that all other work must stop so this can be fixed.  While Lean promotes pulling the Andon cord, I like to point out that there are likely many issues in Quality when Security is involved.  I stop everyone in the moment of “hair-on-fire” and ask them to calm down for a minute.  Breathe.  And then look at the list of prioritized items we agreed to work on.  I ask if this “hair-on-fire” issue should displace any of those issues.  If the answer is yes, we codify it with a risk profile that matches what we did during the Risk Assessment.  If it doesn’t (which is almost always the case), we add it to our master list of “all-the-things-we-should-do” so it’s not forgotten.

Result: You recognize the need to fix issues of Quality, but also to balance that against where the greatest returns are achieved, and how they align with the company’s objectives.  People still feel their concerns are valued, but you also ensure they maintain a balanced and normalized view of priorities.

6. Manage the Flow of Work: This effort was far more ad-hoc in many respects, but I drew on numerous methods and tools for managing work.
   • Make Work Visible: Kanban – nearly every security team I’ve worked with has preferred Kanban as they way to visualize and manage their work. One task in, one task out, pick up the next task.  Because so much of security’s work is intertwined with other teams, it is hard to march to sprint cycles.  We could instead weave in and out of activities – pushing things into “on hold”, and flow with any other style of work more easily.  What we gained was visibility into what was being worked on, and what was yet to be done.

1. • Fit to capacity & Level the Workload – we monitored the Kanban, as well as I had conversations about people’s workload.  If I felt they were being overwhelmed, or if they put in more than 40-45 hours of real work (e.g. I found them in the office after hours all the time), then I would postpone work based on company priorities.  I recognized that quality was going to be the first thing that was sacrificed if I didn’t put things on hold (see Build for Quality).  The team now respected that I valued their sanity, would avoid overwork, and would balance priorities.  They knew they could do the same.  I likewise drove those who didn’t put in the time to deliver.  Taking advantage of my desire for quality was not rewarded, and would be privately confronted.  To quote Nick Galbreath from his time at Etsy: “If you don’t take responsibility, then you probably don’t belong.”

7. Build for Quality:  In many projects I have seen people start with a goal, and a deadline of a date.  I get frustrated by this model because most people are very bad at estimating how long something will take to accomplish.  Even with the concept of an MVP (Minimal Viable Product) they still underestimate the amount of work and time it will require.  To overcome this bias, I lay down a set of rules for every project:
   • Estimate your timeline and amount of work using the worst-case scenario.  We are so over optimistic at time estimation that this will be far more accurate.
   • You are allowed to remove features, but you are not allowed to remove quality.  If the solution will fail to operate shortly after launch, or there is a probability of disruption to regular operations, go back and fix.  Features can be added later.  Quality failures are highly disruptive.
   • Test, Test, Test.  Work on the change, try it, and make the change as many times as you want.  In test (non-prod) environments.  Get good at it.  Make mistakes, practice.  Learn.  Then when you get to production, its close to rote.  You’ve tested all the ways you can think of to fail, and have learned from them for the long term, not just this change.
   • If a deadline is going to be missed, evaluate the cost of doing so.  Then evaluate the cost to the company of pushing it out with the missing quality (e.g. what happens if it fails every other day, if operations stop, if it gives the wrong answers).  Measure this using money and time (which can be equated with money).  Things like loss of customers, financial errors, product failures all have monetary costs that can be measured.  Deadline pushers will give pause.

Result: You will surface over-optimism (it will take time, but you’ll be right more often than the optimists).  You will keep a focus on Quality, and make sure it stays in the forefront.  You’ll encourage learning during testing so that failures are a reward and help avoiding them when they are painful.  You’ll also provide a model for everyone to evaluate the impact of quality versus feature deadlines (there is no correct answer until the measure is made).

Change Technology

By now, I shouldn’t even need to talk anymore.  You should have a team that is on a path to functioning, collaborating, and looking for ways to save time and effort.  They know what they want to do, where they are frustrated, and have surfaced these issues.

Now go Lean.  Find the weaknesses in your flow of work and in your security risks.  Where do you need speed, where do you need to mitigate risk, and where do you need more data.  Examine, communicate, collaborate, and never forget that the Security Team is one piece of an organization that is carefully balancing Opportunity and Risk, and the greatest service the Security Team can deliver is to help guide those Opportunities past the potholes of Risk.