FCAPS – A Quick Introduction

It occurs to me that I’ve been writing the last few posts about network management tasks based on an ITSM model and I didn’t even introduce what is probably the more arguably more useful model for breaking down and understanding network management tasks; the FCAPS model.
FCAPS has it’s roots in the ISO, similar to another model we all know and love; the OSI model. Everyone remember that one? Please Don’t Take Sales’ Peoples Advice? You may have learned another acronym for it, but this is the probably the most basic conceptual model that every networking person uses to understand the world we live in.

For those of you who are looking for some extra credit reading, or need a cure for insomnia, you can find the actual FCAPS standards in the ITU-T M.3400 recommendations. For the rest, I’m hoping to give a brief overview to help you understand the different aspects of the disciplines of network management.

F is for Fault

This involves the detection, isolation, and correction of a fault condition. Or in plain english, this lets you know when things are broken.

Fault Management could involve things like syslog, SNMP traps been escalated to Alarms. Root-Cause-Analysis and Alarm suppression or some AI which tries to seperate the signal from the noise during event storms. Alarm notification policies ( sending out an e-mail once you get an alarm ).

Traditionally this was implemented in a lot of NMSs as Green-is-good management. Basically, if everything is green. Things are ok. If they are yellow or red, you’ve probably got along night ahead of you.

In recent years, Fault Management has started to include application performance management as well. In modern networks, it’s not enough to know that an application is “up”. Now we must also make sure that the level of service, or SLA, that is been delivered to the end-user is adequate to meet their needs.

Note: Whether an activity falls into one category of FCAPS or another might depend on your perspective. If you are measuring bandwidth on a particular port, you may be in the “P”, but if you are measuring the bandwidth and raising an alarm if you cross a certain threshold, you’re now in the “F”.

This may seem confusing at first, but remember that FCAPS is just a conceptual model. This is similar to the 7 Layer OSI model. Ask any good network person what layer MPLS falls at and they will either answer ” It depends” or potentially ” 2.5 “.

C is for Configuration

This involves the configuration of the software and hardware in the network. This includes the versions of software, the actual configurations, change management, etc…

This is probably the easiest to understand. If you’re upgrading code on a switch or router, if you’re logging into a router to make a configuration change, or if you’re just plugging a network cable in to a PC, you’re in the “C”s.

Accounting

This involves the identification of cost to the service provider and payment due for the customer. Ie: Billing.

Personally, I find this definition a little restrictive and prefer to apply the definition that I heard in a presentation. I wish I could remember the name of the gentleman to give him credit. He started out in a thick southern drawl

The thing to remember about a’counting, is that the rest of the world just calls it counting.

I know. Barely funny, right?

But it does allow us to use this to include things like

netflow for counting the different protocols running across a certain WAN link.
SNMP polling of T1/PRI interfaces for ensuring that you’re Erlang calculations are accurate and you don’t need to raise or lower the number of trunks on your voice gateways.
RADIUS to track how long a user was logged into a specific port on the network or how much bandwidth he actually used.

You get the picture. Basically, accounting is just counting things which might be interesting to you.

Although this is not the strict definition from the ITU M.3400, this amended version makes it easier for me to apply this because I don’t have very many customer (read: any) who actually do charge-backs for their services.

Obviously, in a XaaS service, this domain is probably going to get a lot of attention in the coming years.

P is for Performance

This involves evaluating and reporting on the effectiveness of the network, and individual network devices.

Way back when I did my CCNA, one of the things I remember reading about was how you should be checking your routers and switches often to see if their CPU or memory was running high. I’ve never actually met anyone who logged into a device to check on a daily basis, but the advice was actually really good.

With a good NMS, you can

use SNMP polling for the CPU and Memory to track their trending over time.
use ICMP to track availability of the devices ( assuming it responds!)
use ICMP to track the latency of the device to test the quality of the link.

As I mentioned in the Fault section, performance often blurs with fault in that good performance management habits can alert you to faults in the network. In fact, good performance management can even allow you to proactively avoid faults by identifying a potential performance block in the network, and addressing the issue before it turns into a fault.

Probably the most important thing to know about performance management is that it helps you make better decisions.

Most good network engineers can instinctively know where the bottlenecks are in their networks and can usually correctly identify what needs to be upgraded to get the most benefit.

Most great network engineers can use the pretty graphs from a good performance management tool to get the money from their CFO for those upgrades.

In my home network, I actually track the response time of all my links, as well as additional services, such as the one below which allows me to keep my wife happy.

Screen Shot 2012-08-18 at 11.01.45 PM — Facebook Response Time Performance Tracking

note: probably the most recognizable performance management tool would be MRTG/PRTG. I can’t even imagine how many network upgrades were justfied by the pretty graphs that came out of these tools.

Security

Security is… well security. These are the network management activites that involve securing the network and the data running over it.

In a lot of ways, I strongly believe that security should be addressed in every waking (and sleeping!) moment that you’re thinking about your networks. Security should become so second nature to us that it should be almost impossible to perform any of the other tasks without security entering the conversation.

What do I mean?

Fault – CIA – Confidentiality, Availability, and Integrity. Hard to be secure when it’s not available and the Fault domain helps us keep it that way!

Configuration – Auditing – Good configuration management practices can involve automated IT Control objective verification tools, otherwise known as “scripts” which will allow us to have the NMS ensure all the configurations are what they should be and no unneeded services are on our routers and switches.

Performance – You can’t get performance data without SNMP, and if you’re using SNMP, PLEASE USE SNMPv3 if possible! It can be encrypted with integrity. Also, lock down your management interfaces with ACLs on your devices.
FCAPS

It’s just a model

Please don’t take it too seriously. It’s not a binary model. Feel free to apply some fuzzy logic here and be confident that it’s 46% Fault Management and 54% Performance Management.

The important thing here is that it helps us understand the network management world we live in. It gives us a conceptual model to be able to understand the different activities involved in network management. As an added bonus, it also gives us a handy tool to evaluate different NMS software packages.

Think about the tools you’re using. Are you using a point solution, like Solarwinds Orion NPM which focuses on Performance monitoring, or an Open Source tool like RANCID which focuses on Configuration?

Or are you looking at a SPOG solution like HP’s IMC which provides full FCAPS (and more!) in the base package?

What tools are you using? Are they full FCAPS?Or are they more focused on one particular area?

Configuration Management – Software Management

So in the last post I introduced the concepts of the Configuration Management System, and the Configuration Item. Today, I’m going to introduce the concept of the Definitive Media Library.

The DML is really nothing more than a software library. Ideally, this should be tied directly into your element management system so that you can define the baseline software image, deploy the image out to the appropriate devices, and audit the network to ensure that all of the devices are inline with your golden software definitions.

As I laid out in the last post, standardization is there to make your lives easier. But it takes a lot of commitment, especially if your network has gone through significant “organic growth”. Making the choice to commit to good configuration management hygene is sort of like committing to going to the gym or commiting to eat healthier.

Just like going to the gym, the first thing you need to do is figure out your current software state. Hopefully, your NMS software will have the ability to discover and audit the software running on the devices in your network and report against a known good state.

Audit the Current State of the Network

If you don’t have an NCCM tool in place with these features, you may end up writing scripts, or worse case, loging into your devices manually and noting the software version in an excel spreadsheet. Once you have a handle on what’s out there, the next step is chosing what version of code you need to be running.

Choosing your Software Version

So now that you’ve figured out that your devices are all over the place, it’s time to figure out what version of software you actually want to be running. Whether you are running Comware, IOS, NXOS, Junos, FTOS, or some other OS that I haven’t mentioned, the guidelines are pretty much the same.

Wash, Rinse and Repeat.

What about the exceptions?

I was going to try to sugar coat this, but I’ll just come out and say it. Cisco has licensing for many of their platforms, this can create situations where you can’t actually get on a common code version without incurring additional CAPEX costs associated with buying the licenses and OPEX to deal with the SMARTNet’. Or potentially, you can get into the situation where the features you’re looking for are mutually exclusive in two different IOS images for your routers. Or you’re running Cisco Callmanager and your gateways require the Voice image and your regular WAN routers another image.

In any event, my recommendation is still the same. Find the fewest possible combinations of software for the hardware platforms in your network and stick to them unless there is a REALLY good reason to change.

Check out this video of the basic NCCM features in HP’s Intelligent Management Center to help you navigate through your software baseline woes.

Anything I missed here? Feel free to post in the comments below.

Intro to Configuration Management

So in a previous post, I made the recommendation to go find an ITSM framework. For the rest of this series, I’ll be referring to the ITILv3 ITSM framework a lot. The two books that, IMHO, apply the most to Network Operations are the Service Transition and the Service Operations books.

For the next few posts, I’m going to focus on the Service Transition volume, and specifically on the Configuration Management sections.

So in ITILv3, one of the MOST important things to understand is the concept of a Configuration Item.

What’s a CI?

The way I explain this to customers is it’s the smallest managed thing, or set of things, in the environment.

How does that apply to my network?

Well, hopefully, I’m going to try and explain that now.

The first CI in a network might be the hardware devices that are in the network. These are your switches, routers, firewalls, load balancers, servers, etc…

So most people are good with the idea of standardization. It makes senses that it’s easier to manage fewer kinds of devices. This is recommendation #1.

1) Standardize on as few hardware platforms as possible.

The good thing is that this is fairly easy to achieve. In fact a lot of people do this instinctively. They standardize on the same two chassis switches in their core, they use the same model in their distribution, and they use the same model for the access layer.

Here’s where things get crazy though.

Many of the same customers who try to standardize on a current device often have no processes in place to ensure that they are all running the same version of code.

So think back to the ITIL Configuration Item. If you have five HP 5500EI switches, and five different OS’s on them, you now have five different CI’s to track. Make sense?

Five different versions of commands

Five different versions of bugs.

5 times the headache.

If a configuration item is the smallest manageable object, then each of the different combinations of hardware and software count as a single CI. BUT… if we standardize one version of code for that hardware platform, we get one configuration item.

So the first thing that I recommend customers to do is…

GET ON ONE VERSION OF CODE!!!!!!

This is commonly called a golden software version. One version of commands, one version of bugs. One CI.

On the flip side; one of the other common mistakes I see made by customers who have taken the first step of getting on a single version is that of upgrading without a reason.

My recommendation here is to do your homework. When a new version of code is released, read the release notes, check the bug fixes, check the new features. If there’s nothing in there that is addressing an issue you’re having, or new functionality that you NEED to have,

WHY WOULD YOU CHANGE?

It may seem strange, but when you get a new switch out of the box, you may want to just plug that into your network and downgrade it to the older software. More thoughts on this in this blog post.

Any decent NMS should have the ability to be able to define, report, and deploy the correct version of code to the hardware devices.

Funny enough, post writting this, I found ~~this~~ another great blog by Terry Slattery, this time over at http://www.nojitter.com.

What about you guys? What configuration tools are you using? HP IMC? Orion NCM? Rancid? Prime? A TFTP Server on a wandering laptop?

Device Instrumentation

Not all devices are created equal.

I know this seems like a piece of Captn’ Obvious wisdom, but it bears thinking about a little in context of network management.

One of the things which I see all the time is someone asking to do XYZ on the device. Whether that’s pull serial numbers from power supplies, or read the sticker on the back of a switch. There are some things that are just outside the realm of possibility, or would just be to difficult to put into place.

If you are seriously looking at implementing an NMS, you need to get friendly with SNMP. Simple Network Management Protocol is probably the most common management protocol on the planet.

To be honest, SNMP is a second language and I would highly recommend anyone who wants to get SERIOUS about network management pick up a book or two and start learning it. SimpleWeb has some tutorials, podcasts, and slide decks that they make available which may be a good place to start.

In a nutshell, SNMP MIBs fall into two major categories

Public – These are the standard MIBs that are defined by the IETF. These are your friends, the bridge MIB, dot3 MIB, Entity MIB, etc.. MOST vendors should support these.

Private – Also referred to as Enterprise, these are the MIBs which Vendors write to support their own device specific functionality.

Occasionally, someone brings in a non-snmp capable device and asks for it to be monitored. And then they complain because you can’t make the same pretty graphs.

If it’s not instrumented in the device, we can’t do anything with it.

Let me say that again…

If it’s not instrumented in the device, we can’t do anything with it.

Here’s an example: Say someone comes to you and says ” Hey! Can you please tell me what the serial number is on the power supply in XYZ vendors chassis switch?”

I check the MIBs and it seems that XYZ vendor hasn’t instrumented serial numbers as one of the piece of information which they make available. So the answer is ” No, I can’t”

Then they complain that this NMS stuff, or the specific NMS product sucks. Remember

If it’s not instrumented in the device, we can’t do anything with it.

Baselining

Where are you coming from?

One of the first mistakes that new network operators make is that they don’t have a good idea of where they already are.

Take the case above. Say you want to get to Disneyland. Google can tell you were Disneyland is, but if you don’t have a starting point, there’s really no way to understand how you are going to get to where you want to be.
So the first thing you need to do when you’re trying to operate a new network is figure out exactly where you are.

This concept is known as baselining

Baselining:

At its essence, baselining is really nothing more than taking stock of where you are. Most experienced engineers instinctively know when they reach a new environment that they want to spend some time just getting to know the place before they make any changes. Network Management as a discipline takes this to a much more structured level.
There are a few types of baselines, performance and configuration been the major.

Performance Baselines –

This is the simple act of contain and recording things in the network. One of the common mistakes that I see new network management practitioners make is the ” I want to monitor everything! ” move.

Now there are some grounds for this. Remember this guy?

“When you can measure what you are speaking about, and express it in numbers, you know something about it. But when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.”

Lord Kelvin, 1891

Continue reading “Baselining” →

So you want to install an NMS?

So when I started this blog, one of the things I wanted to do was to start putting out some of the knowledge that I’ve gained in network operations over the years. About 5 years ago, I decided that I really wanted to become a network management expert. Except… there was no CCIE Management. So I had to do it the hard way. One of the original reasons of me starting this blog was to try and make that journey a little easier for the next person who has an interest in network management or someone who just gets told they need to implement an NMS project and has no idea where to get started.

Getting started

I know I’m going to take flack for this one from some of the grumpy old network gods, but one of the first places I would HIGHLY recommend visiting on your journey is the ITIL homepage.

The IT Infrastructure Library is essentially a set of “best practices” that have been codified and put on paper. If you’re interested, there’s a intro white paper here. I’ll wait….

So in a nutshell, ITIL is really just an IT Service Management Framework. There are others, like the Microsoft Operational Framework, not to mention FCAPS, eTOM, and probably a few others that bare mentioning, but have slipped my mind.

I’ll admit it, I’m a certification junky, so I went out and got ITIL v3 Foundation certified and what I found out was that ITIL is really just a bunch of common sense best practices that you would learn if you had spent the last 20 years in operations.

For those of us that didn’t, I’ll do a VERY quick and dirty

Intro to ITIL

1) Service Strategy: What should we do?

2) Service Design: How should we do it?

3) Service Transition: Let’s do it.

4) Service Operations: Are we doing it?

5) Continual Process Improvement: Can we do it better?

Yup. 5 volumes. Thousands of pages, and I just reduced it down to 5 lines.

Now there’s a lot more to ITIL and there’s a ton of good information in there around specific things like change control processes, or how to define SLA ( service level agreements ) etc… but IMHO, at it’s very root, it’s just some wisdom written down by people who have been there and done that.

If you’re lucky enough to have an employer pay for your books, I would suggest picking up the last 3 volumes since that’s where the meat of operations actually happens. If you’re not, I would suggest checking out itunes and search for ITIL, ITSM, and start checking out blogs like the IT Sceptic to get a balanced view of the subject.

So step one in the journey to become a network management expert for me was to figure out what actually went in to managing a network. Not just the technical aspects, any CCNA can tell you how to turn on SNMP or to setup SSH on a switch, but I think the first step in this journey is to start to learn about how the business feels about IT Service Management and to start to fill in the other “W”s.

So my first big piece of advice for anyone who wants to really get into network management

Find a Framework

Whether that’s ITIL, or MOF, or your own companies ITSM Methodology, having a framework will provide you with a way to see the world. The only additional piece of caution I would give to you is that it’s just a framework. Use it to help you understand until it’s no longer useful. Remember the OSI model? 7 discrete layers that were written in stone? and then MPLS comes along and suddenly you have to troubleshoot things at layer 2.5.

For ITIL specifically, there are a lot of people who have made a lot of money by preaching it as if it’s gospel. Just smile at them and remember, they are just guidelines. Adjust where you need to.

I think that’s enough for tonight. Hopefully someone finds this useful and I’ll keep motivated to get this out. 🙂

@netmanchris

HP IMC’s New eAPI

Now that I got rum-pooh out of my system… on to a slightly more technical post.

Not sure if anyone of you caught the recent announcements about the new eAPI from HP’s Intelligent Management center. In a nutshell, this is a RESTful API which allows programatic access to almost all ( maybe all?) of the IMC functions through an HTTP(s) interface.

Now I’m not a programer at all, but I like to think I have a working knowledge of programing logic. At least enough to give a half-decent programmer enough information to get the job done.

So when I had a co-worker present me with a problem earlier this week, I thought “Hey, I wonder what this new eAPI can really do?”. I did mention I’m not a programer right? After this little exercise, I’m thinking I might just have to pick up some scripting skills this year. 🙂

So what’s the value of the eAPI? It functionally allows IMC to act as the progamatic upper layer APIs and abstracts the actual management task from the underlying hardware devices.

In less complicated terms; it means that a program can say “Hey, IMC change the VLAN on this port” and IMC, assuming IMC actually supports that particular device, it will change the VLAN on that port.

INDEPENDANT OF THE ACTUAL VENDOR

Yup. That’s right. IMC doesn’t care if that device is a HP 5500EI ( comware ), a HP 3800 ( procurve), or even a Cisco Catalyst 3560. From the perspective of the developer on the other side. It’s as simple as “Hey, IMC change the VLAN on this port”.

So the actual challenge I was given was the following.

” A customer wants to take a bar-code reader, scan in the MAC address of a device, plug it into the network, push a button and then have IMC automatically put it in the right VLAN”.

Now first I had to break that down into the various components.

1) Bar-code reader scans the MAC address

2) Program has to capture the MAC address for use in the %mac-address% variable in the script.

3) Find the device in the network

Hmmm… this could be more difficult than I thought.

So, I need to mock this up, so I break out to a windows CMD prompt.

Ping a known address ( my Synology NAS — LOVE THIS PRODUCT ). And then put do a arp -a to get the following output

10.101.0.51 00-11-32-10-03-8b

Now if I was using the IMC web-interface, I would just use the Resource-Terminal Access-Real-Time Location feature which will, you guessed it, locate a host in real-time using the mac-address or the IP address.

Hmmm… that kinda sucks for output for the script to leverage. So I went and looked at the eAPI documentation and came out with this little baby

The eAPI call is the following ( if you wanted to search for an IP address you would use type=2 instead of 1 )

( Don’t click this, it won’t take you anywhere)

http://10.101.0.201:8080/imcrs/res/access/realtimeLocate?type=1&value=00-11-32-10-03-8b

The return is the following

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

– <list>

– <realtimeLocation>

<locateIp>00:11:32:10:03:8b</locateIp>

<deviceIp>10.101.0.221</deviceIp>

<ifDesc>GigabitEthernet1/0/21</ifDesc>

</realtimeLocation>

</list>

Yup. Good old XML. Easy to apply transforms or grab variables. Programmers love this stuff.

4) So now I have the device IP (switch) that this is plugged into, and the ifDesc which is the actual interface it’s located on. So now I have to figure out how to apply the VLAN to this interface. So I break out the trusty eAPI documentation and start looking for the VLAN section.

Hmmm… I have the devIP and the ifDesc.. not the devID and the ifIndex

note to self: Feedback to the developers to have the first command return the devID and the ifIndex variables

So now I have to find the devID and the ifIndex for that devIP and intDesc

5a) Now if I was on the trusty IMC web interface, I would go to the device resource page… hmmm… that doesn’t appear to be there. I guess instead, let’s go to the eAPI documentation and look for something that looks like a dev query.

Yup. It’s actually called devquery. And it looks like I can filter based on the device IP.

Cool. So now I can search for the specific device IP and hope we get the devID variable back that we need for the VLAN call.

http://10.101.0.201:8080/imcrs/plat/res/device?ip=10.101.0.221

The return is the following

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

– <list>

– <device>

<id>15</id> ———————————————- This is the device ID that we need to reference later

<label>HP_5500EI</label>

<ip>10.101.0.221</ip>

<mask>255.255.255.0</mask>

<status>1</status>

<statusDesc>Normal</statusDesc>

<sysName>HP_E5500EI</sysName>

<contact>HP Montreal</contact>

<location>Marlborough, MA 01752 USA</location>

<sysOid>1.3.6.1.4.1.43.1.16.4.3.36</sysOid>

<sysDescription>HP_5500EI</sysDescription>

<devCategoryImgSrc>Switch</devCategoryImgSrc>

<topoIconName>stack</topoIconName>

<categoryId>1</categoryId>

<symbolId>1022</symbolId>

<symbolName>HP_5500EI</symbolName>

<symbolType>3</symbolType>

<symbolDesc>HP_5500EI</symbolDesc>

<symbolLevel>3</symbolLevel>

<parentId>1003</parentId>

<typeName>3Com S4800G PWR 24-Port</typeName>

<mac>00:1e:c1:dc:fc:01</mac>

<link op=”GET” rel=”self” href=”http://10.101.0.201:8080/imcrs/plat/res/device/15” />

</device>

</list>

There it is. DevID is “15”.

5b) So now we need to figure out that ifIndex value associated with <ifDesc>GigabitEthernet1/0/21</ifDesc> that we pulled above.

If I was in the webinterface, I would simply go to the device ( 10.101.0.221 ), click on the interface list, click on interface Gig 1/0/21 and I would pull out the ifIndex from the interface…

But again, those programmers don’t want HTML, they want an easy XML output that they can play with. So let’s find that…

http://10.101.0.201:8080/imcrs/plat/res/device/15/interface?start=1&size=100

This returns a whole bunch of data for all the interfaces on the switch, but I’m sure that any decent programmer can write a regex expression to only return the one who’s ifDesc value is for Gig 1/0/21, right?

– <interface>

<ifIndex>21</ifIndex> ——————————————–In this case, the ifindex value is the same as the port number. That’s not always going to be true. This is the other variable for the set VLAN

<ifType>6</ifType>

<ifTypeDesc>ETHERNETCSMACD</ifTypeDesc>

<ifDescription>GigabitEthernet1/0/21</ifDescription>

<adminStatus>1</adminStatus>

<adminStatusDesc>Up</adminStatusDesc>

<showStatus>2</showStatus>

<statusDesc>Down</statusDesc>

<operationStatus>2</operationStatus>

<operationStatusDesc>Down</operationStatusDesc>

<ifspeed>10000000</ifspeed>

<appointedSpeed>-1</appointedSpeed>

<ifAlias>GigabitEthernet1/0/15 Interface</ifAlias>

<phyAddress>00:1e:c1:dc:fc:4f</phyAddress>

<mtu>1522</mtu>

<lastChange>4 day(s) 21 hour(s) 39 minute(s) 50 second(s) 990 millisecond(s)</lastChange>

<lastChangeTime>42359099</lastChangeTime>

<filterTrapStatus>0</filterTrapStatus>

</interface>

So now we have the DevID 15 and the ifIndex for the actual interface where that MAC-address is located.

So let’s go back to the set that VLAN

Let’s assume that you wanted to put the device in VLAN 20, you would run the following

http://10.101.0.201:8080/imcrs/vlan/20?devId=15&ifIndex21

That’s about it for the task. Now any decent programmer is going to have to put in some checking and error handling, for instance, you might want to check whether or not that VLAN actually EXISTS on the switch. ( Can’t put a VLAN on a port if the VLAN doesn’t exist on the switch, right? ) or maybe return an error if the MAC-address is actually seen on two interfaces, but in a nutshell that’s it.

note: I would also suggest that the dev actually bounce the port to make sure that the device hasnt’ gotten locked in with a DHCP address on the wrong subnet.

/plat/res/device/{deviceId}/interface/{ifIndex}/down to down the interface

so for use that would be /plat/res/device/15/interface/21/down

and then immediately do a

/plat/res/device/{deviceId}/interface/{ifIndex}/up

or again for us /plat/res/device/15/21/up

Now whether the actual switch commands are “switchport acces vlan 20 ” or ” port access vlan 20 ” or some other variation on a theme doesn’t actually matter to your devOps team. They just write the code to follow the steps and IMC and the eAPI will take care of the rest.

Pretty cool stuff. 🙂

@netmanchris