So it’s obvious I’m a management guy. And I don’t mean I manage people, I manage my network, and I help advise my customers on managing their networks. One of the things that never ceases to amaze me is how little people in our industry actually know about change management. In fact I find it down right ironic that an industry who’s only constant is change hasn’t really  embraced change management at all. In fact, I think it’s safe to say we run screaming from change management as quickly as possible. The only thing we avoid more than change management is it’s dreaded enforcer

THE PROJECT MANAGER

Personally, I think that a little more time spent in ITSM ( IT service management ) school could really do a lot to change the stereotype of our profession as networking focused IT workers. The most common ITSM framework today is, of course, ITIL.  The Information Technology Infrastructure Library is a CBOK ( common body of knowledge ) that focuses on how to run an IT organization. It’s most lofty goal is to give us a common vocabulary to be able to effectively communicate across the IT silos. It’s as simple as that. The focus of ITIL is really just to standardize and codify the best practices and wisdom that other IT professionals have gathered over the years and allow us to leverage their prior experience.

def: WISDOM – The experience we get from making bad decisions.

So what does ITIL have to do with code and my wife? We’re getting there, but first we need a little more ITIL knowledge.

ITIL is comprised of 5 core volumes, ( Strategy, Design, Transition, Operations, and CSI for those of you who are counting ).  There are a couple of volumes, transition and operations, that deal specifically with change management. There’s a LOT of good content in there, but I’m going to focus on just one piece

def: Configuration Item: A CI is an asset, service component or other item that is, or will be, under the control of Configuration Management. CI’s may vary widely in complexity, size and type, ranging from an entire service or system including all hardware, software, documentation and support staff to a single software module or a minor hardware component. Configuration items may be grouped and managed together, e.g. a set of components may be grouped into a release. CI’s should be selected using established selection criteria, grouped, classified and identified in such a way that they are manageable and traceable through the service lifecycle.

One of the goals of Configuration Management is really to offer stability in the IT environment. One of the ways to get more stability is to ensure consistency of operations, and one of the way to ensure consistency of operations is to ensure that you have common components. This is way telco’s often standardize on a single box, even though in many cases it may be vastly overkill for the application. From an operational cost point of view, the CAPEX cost is offset by the stability and consistency gained by having a single common CI.

So let me bring this back to my wife…  One of the MOST common mistakes  inefficiencies that I see in many customers networks is the following.

Joe Admin receives a brand new Cisco 3750/Juniper EX4200/HP 5500EI switch for the new wiring closet. It arrived on time, and he got a good price on it. In fact he got a great price because it’s the exact same model that he’s been buying now for a couple of years. So Joe grabs his trusty serial-to-usb adapter, grabs his MacBook Pro and console cable and 10 minutes later, the box is assigned an IP address to VLAN interface 1 and he’s setup telnet and a local user account for remote administration purposes here and he’s good to go!

Now there are a bunch of things that he’s just done that are less than ideal  (see if you can spot the all! ) but I’m just going to focus on one today; Joe didn’t check the version of code on this switch.  And this brings me back to my point.

Code is like my wife

Let me try to explain that one: Every version of code has things that we love, things that we don’t. It has features that we may or may not use, and always, always, always bugs. Of course vendors do their best to ensure that code is as bug free as possible before it goes out the door, but there is always something unintended or unexpected that comes up.

I’ve been married for a few years now, and like any married person I’ve found that my wife has certain… special charms that make me love her even more.   I’m not allowed to call them bugs. But there are certain things in peoples personalities that are just like bugs in code. You can’t predict them, they really don’t seem to make any sense, and they usually pop up at the LEAST desirable moment.

So how is code like my wife? Both have bugs of course! ( I mean undocumented “features”!!!)  But…  we’ve been together for a few years now and I’ve got a lot of the bugs figured out. I understand that when I do X she will exhibit a certain symptom that may not make sense to me, but I can tell you that it’s repeatable on a somewhat consistent basis. ( Yes, I’m male. So that does make me a slow learner ).

So let’s apply that back to Joe.  Joe just grabbed a brand new switch out of the box, with a brand new software load and a brand-new set of undocumented “feature” that he has yet to discover. Now there are certain people in the world who enjoy the excitement of new experiences. The thrill of discovery. The rush of jumping headfirst off a cliff with no clue whether or not there are rocks in that water. Me? Not so much.

I love my wife, but I have to admit that the practical engineer side of me really loves the stability as well. I know a lot of her bugs. I know what sets her off. I know how to avoid them, and when they happen, I know how to diffuse them quickly. I also know that if I have to troubleshoot for a new bug, I have a wide range of experience with this particular version to draw on, which should speed up the whole MTTR (mean time to recovery) process.

Note: I usually use MTTI ( Mean Time to Innocense) , but in my experience so far with my wife has taught me that it’s VERY rarely not me who’s at fault.

So What Should Joe have done here?

1) Pick a version of code. Any version of code. but only pick one. Get to know it inside and out. Read the configuration guide, the release notes and make sure he’s happy with the features and documented bugs.

2) Install THIS specific version of code on every single compatible hardware platform in the entire network.

3) When Joe receives a new switch with a new version of code. Read the release notes. Check for the new features. Are they required? Is there a solid business reason for upgrading? Did the code fix any specific bugs which are impacting the availability of the network or the profitability of the business?   If the answer is No to all of these questions.

DOWNGRADE THE SWITCH!

Yup. You heard me, I’m recommending that you actually downgrade your switches to the chosen version of code. Your knee-jerk reaction would be to upgrade everything else in the network to the new version, right? Wrong.

Wrong. Wrong. Wrong. Wrong. Wrong.

In a industry where there is so much change on a daily basis, we often don’t take the time to actually sit down and think about WHY we are changing thing in the environment. What’s the net benefit for this change? Are we changingg just for changes sake? Or is there a compelling reason for us to invest the time and energy into learning an entire new set of bugs? To link this back to the ITIL CI, if you have a single hardware platform with a single version of code on it, you have a single CI. The fewer CI’s you have to manage, the fewer CIs you have to manage. And I think we can all agree that eliminating complexity and reducing the number of managed elements we have to think about in a day is a good thing.

 

@netmanchris