I recently had a pair of encounters that have opened my eyes further to both the causes of our current messy state of IT affairs, and given me hope for a better future. In both cases the issue that came up with access to production environments.
In one particular case a user had their access removed – ostensibly on the grounds that their access violated “segregation of duties between development and production”. There are numerous control frameworks that demand a segregation of production and development environments. There are even others that say personnel should be fully segregated. Lets look at where this came from, and what the outcome has been:
- Segregation of duties came about as a control for preventing one person from performing an end-to-end activity by introducing a check that the activity was appropriate. It started largely as a financial control. The most obvious is preventing an Accounts Payable clerk from inputting a purchase or payment request, and then processing that payment request themselves – all for the benefit of their own personal bank account.
- This control was extended to IT – especially during the Sarbanes-Oxley days – as a way to ensure that a developer could not introduce ways into the programs to siphon off pennies all for the benefit of their own personal bank account.
- This control was then extended further to include personnel access to anything in production because (again ostensibly) it was believed that sharing information about production would create knowledge that developers could exploit.
Lets be clear. Controls that prevent the theft of money (fraud) are important. However the lengths to which this control has been extended has become ludicrous. What it has done is damage the workflow, trust, collaboration and functioning of the IT department and its ability to support the business needs of all other parts of the company. How you ask?
- The segregation-of-duties controls are extended to deny developers visibility into the environment, which means their situational awareness of how their programs are running is removed.
- They lose the belief that other groups trust them since their visibility is removed. They pull up a wall.
- They now view the operation of a program as “someone else’s problem since they don’t let us in”. The pull that wall up higher.
- They now throw programs over the wall – because “we’re not responsible for them in operations”. Operations hates when this happens.
- Myriads of other controls flow in to stop-gap the problems that development teams don’t have the visibility to understand. Testing requirements increase to address the problems since it is believed the problem is in insufficient testing. The testing becomes cumbersome, laborious, and yet largely ignorant to the problems that happen in production.
- Costs go up, blame goes up, and failures happen…and the speed of work goes down.
Sounding familiar yet?
So how does this fit into my realization? Access into production for developers is not a bad thing. Developers should have visibility into application and system logs so they can view the reaction of their code in real world situations. Developers should have the ability to see elements that are not sensitive. They likely shouldn’t see sensitive data like payment cards, or encryption keys, but they should be able to see configuration files, data types and definitions. Give developers what they need to create a feedback loop that is clear, unobstructed, but doesn’t violate regulations.
That being said, developers promoting code into production without checks and balances is a bad thing. That I think we can agree on, but how does that fly with a DevOps mentality? How about:
- Changes can go into production once they go through an automated test suite. They are only available for check-out when they meet that criteria of that automated test suite.
- Production personnel (ops) can promote into production anything that has gone through the test suite and is available for check-out into production.
- Development personnel can check problems and push fixes through this same chain.
If you notice, in the better world, developers have access to view, and monitor the production environment – they have a feedback loop. In the better world, developers still have to have their programs vetted by a testing procedure before changes are pushed to production. The key control objective is still met – reduce the probability for fraud – but with controls that keep the collaboration, accountability, and teamwork in place.
Now in the two cases that I came across, both arrived at the same conclusion. Both believed that visibility was important. Both believed that it could be achieved. The challenge was to educate those who have accepted the de facto standard of full segregation without understanding the original goal, and the impact of such a decision.
