I had the opportunity to attend the Open Networking User Group event (ONUG) in New York recently and had a chance to talk through some of my musings around change management in an SDN world with some very smart, knowledgable people from a range of different backgrounds.
Let’s talk a little about change control
In a nutshell, people screw things up when left to their own devices. Individuals will inevitably type a wrong command, misplace a decimal point, not have sufficient information, or just plain not-think-something-through.
People are frail, fragile and error prone. But when people come together in groups, share information, share experience, and double check each other’s work, then the error-rate per change tends to drop significantly and changes start to be implemented in a much higher quality fashion.
Change Policy in Modern Organizations
Most modern organizations have some change management process in place. Whether they have succumbed to a full ITIL based process, gone the DevOps route of continual integration, or fall somewhere in between, people have generally figured out that change management is a good thing.
I’ve seen good change management that promotes healthy growth, and I’ve seen bad change managements that restricts the business into stagnation because nothing is ever allowed to change in the organization. ( There’s another word for something that never changes – dead. 🙂
Change Control in a SDN environment
One of the major issues I see in SDN environments is that many of the changes that we are not only capable, but advocating, are currently heavily restricted through the existing organizations change policies.
To make this example more concrete, let’s talk about an app from Guardicore that uses SDN to detect potential advanced persistent threat attacks in the data center and then uses OpenFlow and the HP VAN SDN Controller to dynamically keep the session alive and re-route ( re-bridge?) the flow directly to a Honeypot which is capable of performing further analysis on that particular session to see if that particular flow is trying to do anything more interesting like trying to execute shell code or some other dubious shenanigans.
Now imagine how the Change Advisory Board is going to react to this request. I imagine it could go something like this.
” What? You want to reconfigure the edge, distribution and core of my data center based on an unknown event at an unknown time because something may or may not be going on?”
How do you think that’s going to go?
ITSM Pre-Approved Changes
There is a concept in ITSM frameworks like ITIL and MOF that allow for a common change to be pre-approved. The change request still has to be fed into the system, but the approvals are automatic and no one has to actively log into a system and click the ” I Approve “
One of the approaches I’ve been advocating is the possibility of repurposing the pre-approved change to allow for dynamic flow modification based on known conditions. This seems to be the simplest way for us to allow the ITSM structures in well-run IT organizations to continue to work without having to scrape the whole change approval process.
This is new ground and I think that this topic requires a lot more discussion that we are currently giving it.
What do you think? Is pre-approved change the way to go? Is there another better way? Is your organization currently using SDN and found a way to rationalize this to the Change Advisory Board?
Please blog it up or post in the comments below.
kontrolissues 
I think you might be over-thinking it.
Consider a typical network using dynamic routing protocols – do you need a change (pre-approved or otherwise) for the network to react to changing conditions, such as a link flap? No, because that’s the network doing what it’s supposed to do. The Change Request comes when you implement that network, not when it works as part of normal operation.
Another way of looking at it is like DRS. The change is “Implement DRS”. The normal operation is to then move VMs around dynamically, in response to changing conditions.
Are you actually *changing* the network/system configuration, or is it just “working as designed” ?
As an aside, I think that the organisations that are desperate to cling to ITIL processes are a very long way away from implementing any form of SDN.
I thought I was overthinking it as well until I spoke with a customer who was actually in this exact situation. I think your comment ” Are you actually *changing* the network/system configuration or is it just “working as designed ” is spot on.
The real challenge here is getting people to think differently about their environment, configurations and change in general.
Here’s a question for you though; does this perspective/train-of-thought change if we were talking programtic NMS here instead of OpenFlow controller-sourced flowmods?
Does the method used to make the change actually impact how we perceive the change?
Hmmm.. need to think on that one a bit more.
@netmanchris
“Does the method used to make the change actually impact how we perceive the change?”
Possibly. Obviously if it’s people manually making the change, there’s a higher risk of fat-fingering.
Where it gets interesting is how different organisations will perceive it – some orgs are quite happy to write their own scripts to do things, and automate changes. Other orgs wouldn’t dream of using a script – but if you give them a commercial product, and slap an interface on it, they’re happy (even though it’s only really some scripting underneath).
Smarter customers won’t care too much about the exact method, but they will be interested in things like failure detection, rollback, success detection, etc – basically “how robust is this setup, or is a fragile screen-scraping monstrosity that only works with a very specific code release?”
Agree 100%. I was chatting with some of the SoMe tweeps after ONUG and I think we all agree, 2014 is going to be a VERY interesting year.
Can’t wait to see how it plays out!