Using JSONSchema to Validate input

There are a lot of REST APIs out there. Quite a few of them use JSON as the data structure which allows us to get data in and out of these devices. There are a lot of network focused blogs that detail how to send and receive data in and out of these devices, but I wasn’t able to find anything that specifically talked about validating the input and output of the data to make sure we’re sending and receiving the expected information.

Testing is a crucial, and IMO too often overlooked, part of the Infrastructure as Code movement. Hopefully this post will help others start to think more about validating input and output of these APIs, or at the very least, spend just a little more time thinking about testing your API interactions before you decide to automate the massive explosion of your infrastructure with a poorly tested script. 🙂

What is JSONSchema

I’m assuming that you already know what JSON is, so let’s skip directly to talking about JsonSchema. This is a pythonlibrary which allows you to take your input/output and verify it against a known schema which defined the data types you’re expecting to see.

For example, consider this snippet of a schema which defines what a valid VLAN object looks like

"vlan_id":
{
    "description": "The unique ID of the VLAN. IEEE 802.1Q VLAN identifier (VID)", 
    "minimum": 1, 
    "maximum": 4094, 
    "type": "integer", 
    "sql.not_null": true
}

You can see that this is a small set of rules that defines what is a valid entry for the vlan_id property of a VLAN. As a network professional, it’s obvious to us that a valid VLAN ID must be between 1 and 4094. We know this because we deal with it every day. But our software doesn’t know this. We have to teach it what a valid vlan_id property looks like and that’s what the schema does.

our software doesn’t know this. We have to teach it

Why do we care?

Testing is SUPER important. By being able to test the input/output of what you’re feeding into your automation/orchestration framework, it can help you to avoid, at worst, a total meltdown, or, at best, a couple of hours trying to figure out why your code doesn’t work.

Using JSONSchema

So the two things you’re going to need to use JSONSchema are

The JSON Schema for a specific API endpoint
The input/output that you want to validate.

In this case, we’ll use a VLAN object that is coming out of an ArubaOS-Switch REST API.

You did know the ArubaOS-Switches have a REST API, right?

Step1 – Loading the VLAN object

We’re going to gather the output from the VLANS API. Instead of writing custom code, we’ll just use the pyarubaoss library. I’ll leave you to check out the GitHub repo and just paste the contents of the output of a single VLAN JSON output here. I’m also going to create a second VLAN with a VLAN_ID of 5000. Just to show how this works. 5000 of course, is not valid and we’d like to prove that. Right?

Step 2 – Loading the JSON Schema Definition

Now we have the output, we want to make sure that the output here complies with the JSON Schema definition that we’ve been provided.

Loading the JSON schema

Here’s a sub-set of the JSON schema that defines what a valid VLAN looks like

Step 3 – Importing the JSON Schema Library and Validating

Now we’re going to load the JSON Schema library into our python session and use it to validate the VLAN object using the Schema we defined above.

First we’ll look at the vlan_good object and run it through the validate function

As you can see, there’s nothing to see here. Basically this means that the vlan_good object is conforming properly to the provided JSON Schema. The VLAN ID is valid as it’s a integer value between 1 and 4094

Now let’s take a look at the vlan_bad object and run it through the same validate function

We can see that the validate function now raises an exception because and let’s us know very specifically that the VLAN ID 5000 is not valid

jsonschema.exceptions.ValidationError: 5000is greater than the maximum of 4094

Pretty cool right? We can still definitely shoot ourselves in the foot, but at least we know the input/output data that we’re using for our API is valid. To me this is important for two reasons

I can validate that the data I’m trying to send to a given API conforms to what that API is expecting
I can validate that the vendor didn’t suddenly change their API which is going to break my code

Wrap Up

There are a lot of networking folk who have started to take on the new set of skills required to automate their infrastructure. One of the most crucial parts of this is testing and validating the code to ensure that you’re not just blowing up your network more efficiently. JSON Schema is one of the tools in your tool box that can help you do that.

Comments, questions? Let me know

@netmanchris

Shedding the Lights on Operations: REST, a NMS and a Lightbulb

It’s obvious I’ve caught the automation bug. Beyond just automating the network I’ve finally started to dip my toes in the home automation pool as well.

The latest addition to the home project was the Philipps hue light bulbs. Basically, I just wanted a new toy, but imagine my delight when I found that there’s a full REST API available.

I’ve got a REST API and a light bulb and suddenly I was inspired!

The Project

Network Management Systems have long suffered from information overload.

Notifications have to be tuned and if you’re really good you can eventually get the stream down to a dull roar. Unfortunately, the notification process is still broken in that the notifications are generally dumped into your email which if you are anything like me…

NewImage

Yes. That’s really my number as of this writing

One of the ways of dealing with the deluge is to use a different medium to deliver the message. Many NMS systems, including HPE IMC, has the capability of issuing audio alarms, but let’s be honest. That can get pretty annoying as well and it’s pretty easy to mute them.

I decided that I would use the REST interfaces of the HPE IMC NMS and the Phillips Hue lightbulbs to provide a visual indication of the general state of the system.Yes, there’s a valid business justifiable reason for doing this. But c’mon, we’re friends? The real reason I worked on this was because they both have REST APIs and I was bored. So why not, right?

The other great thing about this is that you don’t need to spend your day looking at a NOC screen. You can login when the light goes to whatever color you decide is bad.

Getting Started with Philipps Hue API

The Philipps SDK getting started was actually really easy to work through. As well, there’s an embedded HTML interface that allows you to play around with the REST API directly on the hue bridge.

Once you’ve setup your initial authentication to the bridge ( see the getting started guide ) you can login to the bridge at http://ip_address/debug/clip.html

From there it’s all fun and games. For instance, if you wanted to see the state of light number 14, you would navigate to api/%app_name%/lights/14 and you would get back the following in nice easy to read JSON.

http://ipaddress/debug/clip.html/

NewImage

From here, it would be fairly easy to use a http library like REQUESTS to start issuing HTTP commands at the bridge but, as I’m sure you’re aware by now, there’s very little unread territory in the land of python.

PHUE library

Of course someone has been here before me and has written a nice library that works with both python 2 and python 3. You can see the library source code here, or you can simple

>>> pip install phue

From your terminal.

The Proof of Concept

You can check out the code for the proof of concept here. Or you can watch the video below.

Breaking down the code

1) Grab Current Alarm List

2) Iterate over the Alarms and find the one with the most severe alarm state

3) Create a function to correlate the alarm state to the color of the Philipps Hue lightbulb.

4) Wait for things to move away from green.

Lessons Learned

The biggest lesson here was that colours on a screen and colours on a light bulb don’t translate very well. The green and the yellow lights weren’t far enough apart to be useful as a visual indicator of the health of the network, at least not IMHO.

The other thing I learned is that you can waste a lot of time working on aesthetics. Because I was leveraging the PHUE library and the PYHPEIMC library, 99% of the code was already written. The project probably took me less than 10 minutes to get the logic together and more than a few hours playing around with different colour combinations to get something that I was at least somewhat ok with. I imagine the setting and the ambient light would very much effect whether or not this looks good in your place of business.If you use my code, you’ll want to tinker with it.

Where to Next

We see IoT devices all over in our personal lives, but it’s interesting to me that I could set up a visual indicator for a NOC environment on network health state for less than 100$. Just thinking about some of the possibilities here

Connect each NOC agents ticket queue with the light color. Once they are assigned a ticket, they go orange for DO-NOT-DISTURB
Connect the APP to a Clearpass authentication API and Flash the bulbs blue when the boss walks in the building. Always good to know when you should be shutting down solitaire and look like you’re doing something useful, right?
Connect the APP to a Meridian location API and turn all the lights green when the boss walks on the floor.

Now I’m not advocating you should hide things from your boss, but imagine how much faster network outages would get fixed if we didn’t have to stop fixing them to explain to our boss what was happening and what we were going to be doing to fix them, right?

Hopefully, this will have inspired someone to take the leap and try something out,

Comments, questions?

@netmanchris

Serial numbers how I love thee…

No one really like serial numbers, but keeping track of them is one of the “brushing your teeth” activities that everyone needs to take care of. It’s like eating your brussel sprouts. Or listening to your mom. You’re just better of if you do it quickly as it just gets more painful over time.

Not only is it just good hygene, but you may be subject to regulations, like eRate in the United States where you have to be able to report on the location of any device by serial number at any point in time.

Trust me, having to play hide-and-go seek with an SSH session is not something you want to do when government auditors are looking for answers.

I’m sure you’ve already guessed what I’m about to say, but I”ll say it anyway…

There’s an API for that!!!

HPE IMC base platform has a great network assets function that automatically gathers all the details of your various devices, assuming of course they supportRFC 4133, otherwise known as the Entity MIB. On the bright side, most vendors have chosen to support this standards based MIB, so chances are you’re in good shape.

And if they don’t support it, they really should. You should ask them. Ok?

So without further ado, let’s get started.

Importing the required libraries

I’m sure you’re getting used to this part, but it’s import to know where to look for these different functions. In this case, we’re going to look at a new library that is specifically designed to deal with network assets, including serial numbers.

In [1]:

from pyhpeimc.auth import *
from pyhpeimc.plat.netassets import *
import csv

In [2]:

auth = IMCAuth("http://", "10.101.0.203", "8080", "admin", "admin")

In [3]:

ciscorouter = get_dev_asset_details('10.101.0.1', auth.creds, auth.url)

How many assets in a Cisco Router?

As some of you may have heard, HPE IMC is a multi-vendor tool and offers support for many of the common devices you’ll see in your daily travels.

In this example, we’re going to use a Cisco 2811 router to showcase the basic function.

Routers, like chassis switches have multiple components. As any one who’s ever been the ~~victem~~ owner of a Smartnet contract, you’ll know that you have individual components which have serial numbers as well and all of them have to be reported for them to be covered. So let’s see if we managed to grab all of those by first checking out how many individual items we got back in the asset list for this cisco router.

In [4]:

len(ciscorouter)

Out[4]:

What’s in the box???

Now we know that we’ve got an idea of how many assets are in here, let’s take a look to see exactly what’s in one of the asset records to see if there’s anything useful in here.

In [5]:

ciscorouter[0]

Out[5]:

{'alias': '',
 'asset': 'http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=1',
 'assetNumber': '',
 'boardNum': 'FHK1119F1DX',
 'bom': '',
 'buildInfo': '',
 'cleiCode': '',
 'containedIn': '0',
 'desc': '2811 chassis',
 'devId': '15',
 'deviceIp': '10.101.0.1',
 'deviceName': 'router.lab.local',
 'firmwareVersion': 'System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)',
 'hardVersion': 'V04 ',
 'isFRU': '2',
 'mfgName': 'Cisco',
 'model': 'CISCO2811',
 'name': '2811 chassis',
 'phyClass': '3',
 'phyIndex': '1',
 'physicalFlag': '0',
 'relPos': '-1',
 'remark': '',
 'serialNum': 'FHK1119F1DX',
 'serverDate': '2016-01-26T15:20:40-05:00',
 'softVersion': '15.1(4)M, RELEASE SOFTWARE (fc1)',
 'vendorType': '1.3.6.1.4.1.9.12.3.1.3.436'}

What can we do with this?

With some basic python string manipulation we could easily print out some of the attributes that we want into what could easily turn into a nicely formated report.

Again realise that the example below is just a subset of what’s available in the JSON above. If you want more, just add it to the list.

In [7]:

for i in ciscorouter:
    print ("Device Name: " + i['deviceName'] + " Device Model: " + i['model'] +
           "\nAsset Name is: " + i['name'] + " Asset Serial Number is: " +
           i['serialNum']+ "\n")

Device Name: router.lab.local Device Model: CISCO2811
Asset Name is: 2811 chassis Asset Serial Number is: FHK1119F1DX

Device Name: router.lab.local Device Model: VIC2-2FXO
Asset Name is: 2nd generation two port FXO voice interface daughtercard on Slot 0 SubSlot 2 Asset Serial Number is: FOC11063NZ4

Device Name: router.lab.local Device Model:
Asset Name is: 40GB IDE Disc Daughter Card on Slot 1 SubSlot 0 Asset Serial Number is: FOC11163P04

Device Name: router.lab.local Device Model:
Asset Name is: AIM Container Slot 0 Asset Serial Number is:

Device Name: router.lab.local Device Model:
Asset Name is: AIM Container Slot 1 Asset Serial Number is:

Device Name: router.lab.local Device Model:
Asset Name is: C2811 Chassis Slot 0 Asset Serial Number is:

Device Name: router.lab.local Device Model:
Asset Name is: C2811 Chassis Slot 1 Asset Serial Number is:

Why not just write that to disk?

Although we could go directly to the formated report without a lot of extra work, we would be losing a lot of data which we may have use for later. Instead why don’t we export all the available data from the JSON above into a CSV file which can be later opened in your favourite spreadsheet viewer and manipulated to your hearst content.

Pretty cool, no?

In [9]:

keys = ciscorouter[0].keys()
with open('ciscorouter.csv', 'w') as file:
    dict_writer = csv.DictWriter(file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(ciscorouter)

Reading it back

Now we’ll read it back from disk to make sure it worked properly. When working with data like this, I find it useful to think about who’s going to be consuming the data. For example, when looking at this remember this is a CSV file which can be easily opened in python, or something like Microsoft Excel to manipuate further. It’s not realy intended to be read by human beings in this particular format. You’ll need another program to consume and munge the data first to turn it into something human consumable.

In [12]:

with open('ciscorouter.csv') as file:
    print (file.read())

firmwareVersion,vendorType,phyIndex,relPos,boardNum,phyClass,softVersion,serverDate,isFRU,alias,bom,physicalFlag,deviceName,deviceIp,containedIn,cleiCode,mfgName,desc,name,hardVersion,remark,asset,model,assetNumber,serialNum,buildInfo,devId
"System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)",1.3.6.1.4.1.9.12.3.1.3.436,1,-1,FHK1119F1DX,3,"15.1(4)M, RELEASE SOFTWARE (fc1)",2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,0,,Cisco,2811 chassis,2811 chassis,V04 ,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=1,CISCO2811,,FHK1119F1DX,,15
,1.3.6.1.4.1.9.12.3.1.9.3.114,14,0,FOC11063NZ4,9,,2016-01-26T15:20:40-05:00,1,,,2,router.lab.local,10.101.0.1,13,,Cisco,2nd generation two port FXO voice interface daughtercard,2nd generation two port FXO voice interface daughtercard on Slot 0 SubSlot 2,V01 ,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=14,VIC2-2FXO,,FOC11063NZ4,,15
,1.3.6.1.4.1.9.12.3.1.9.15.25,30,0,FOC11163P04,9,,2016-01-26T15:20:40-05:00,1,,,2,router.lab.local,10.101.0.1,29,,Cisco,40GB IDE Disc Daughter Card,40GB IDE Disc Daughter Card on Slot 1 SubSlot 0,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=30, ,,FOC11163P04,,15
,1.3.6.1.4.1.9.12.3.1.5.2,25,6,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,3,,Cisco,AIM Container Slot 0,AIM Container Slot 0,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=25,,,,,15
,1.3.6.1.4.1.9.12.3.1.5.2,26,7,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,3,,Cisco,AIM Container Slot 1,AIM Container Slot 1,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=26,,,,,15
,1.3.6.1.4.1.9.12.3.1.5.1,2,0,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,1,,Cisco,C2811 Chassis Slot,C2811 Chassis Slot 0,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=2,,,,,15
,1.3.6.1.4.1.9.12.3.1.5.1,27,1,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,1,,Cisco,C2811 Chassis Slot,C2811 Chassis Slot 1,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=27,,,,,15

What about all my serial numbers at once?

That’s a great question! I’m glad you asked. One of the most beautiful things about learning to automate things like asset gathering through an API is that it’s often not much more work to do something 1000 times than it is to do it a single time.

This time instead of using the get_dev_asset_details function that we used above which gets us all the assets associated with a single device, let’s grab ALL the devices at once.

In [13]:

all_assets = get_dev_asset_details_all(auth.creds, auth.url)

In [14]:

len (all_assets)

Out[14]:

That’s a lot of assets!

Exactly why we automate things. Now let’s write the all_assets list to disk as well.

**note for reasons unknown to me at this time, although the majority of the assets have 27 differnet fields, a few of them actually have 28 different attributes. Something I’ll have to dig into later.

In [15]:

keys = all_assets[0].keys()
with open('all_assets.csv', 'w') as file:
    dict_writer = csv.DictWriter(file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(all_assets)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-e4c553049911> in <module>()
 3     dict_writer = csv.DictWriter(file, keys)
 4     dict_writer.writeheader()
----> 5dict_writer.writerows(all_assets)

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/csv.py in writerows(self, rowdicts)
 156         rows = []
 157         for rowdict in rowdicts:
--> 158rows.append(self._dict_to_list(rowdict))
 159         return self.writer.writerows(rows)
 160

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/csv.py in _dict_to_list(self, rowdict)
 147             if wrong_fields:
 148                 raise ValueError("dict contains fields not in fieldnames: "
--> 149 + ", ".join([repr(x) for x in wrong_fields]))  150         return [rowdict.get(key, self.restval) for key in self.fieldnames]
 151

ValueError: dict contains fields not in fieldnames: 'beginDate'

Well That’s not good….

So it looks like there are a few network assets that have a different number of attributes than the first one in the list. We’ll write some quick code to figure out how big of a problem this is.

In [16]:

print ("The length of the first items keys is " + str(len(keys)))
for i in all_assets:
    if len(i) != len(all_assets[0].keys()):
       print ("The length of index " + str(all_assets.index(i)) + " is " + str(len(i.keys())))

The length of the first items keys is 27
The length of index 39 is 28
The length of index 41 is 28
The length of index 42 is 28
The length of index 474 is 28
The length of index 497 is 28
The length of index 569 is 28
The length of index 570 is 28
The length of index 585 is 28
The length of index 604 is 28
The length of index 605 is 28
The length of index 879 is 28
The length of index 880 is 28
The length of index 881 is 28
The length of index 882 is 28
The length of index 883 is 28
The length of index 884 is 28
The length of index 885 is 28
The length of index 886 is 28

Well that’s not so bad

It looks like the items which don’t have exactly 27 attribues have exactly 28 attributes. So we’ll just pick one of the longer ones to use as the headers for our CSV file and then run the script again.

For this one, I’m going to ask you to trust me that the file is on disk and save us all the trouble of having to print out 1013 seperate assets into this blog post.

In [18]:

keys = all_assets[879].keys()
with open ('all_assets.csv', 'w') as file:
    dict_writer = csv.DictWriter(file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(all_assets)

What’s next?

So now that we’ve got all of our assets into a CSV file which is easily consumable by something like Excel, you can now chose what to do with the data.

For me it’s interesting to see how vendors internally instrument their boxes. Some have serial numbers on power supplies and fans, some don’t. Some use the standard way of doing things. Some don’t.

From an operations perspective, not all gear is created equal and it’s nice to understand what’s supported when trying to make a purchasing choice for something you’re going to have to live with for the next few years.

If you’re looking at your annual SMARTnet upgrade, at least you’ve now got a way to easily audit all of your discovered environment and figure out what line cards need to be tied to a particualr contract.

Or you could just look at another vendor who makes your life easier. Entirely your choice.

@netmanchris

Automating your NMS build – Part 4 Adding Custom Views

This is part four in a series of using python and the RESTful API to automate the configuration of the HP IMC network management station. Like with all things, it’s easier to learn something when you’re able to find a good reason to use those skills. I decided to extend my python skills by figuring out how to use python to configure my NMS using the RESTful API.

If you’re interested, check out the other posts in this series

Creating Operators

Adding Devices

Changing Device Categories

Adding Custom Views

In this post, we’re going to use the RESTful API to programatically add a custom view, and then add devices to that custom view. For those of you who don’t know, a custom view is simply a logical grouping of devices. In IMC custom views also form the basis of the topology maps. In fact, a custom view and a topology map are essentially the same object internally, The difference is just whether you chose to look at it as a list of devices in the normal interface or chose to look at it in the typical topology/visio format which we all know and might-not-love.

Why might we want to add a custom view programatically you ask? While, the answer to that might be simply that we’re lazy and it’s easier and faster. Or it might be that you don’t want to have to look through all the different devices, or, as in my case, that you simply want an excuse to extend your python skills. Whatever you’re reasons are, custom views are something that just don’t get used enough in my opinion.

Custom views are a great way to be able to zoom in on the status of a specific branch, a geographic area, maybe a logical grouping of devices that support a specific application? It really doesn’t matter and the best part is that a single device can exist in multiple custom views at the same time, so there’s really no limit to how you put these views together. It all depends on what makes sense to you.

The Code

This code is a little bit more complicated than some of the other examples we’ve looked at. We’re actually going to be using multiple functions together, but the logic should be pretty easy to follow. I’m sure there are better ways to do this, but this seems to work for me. I’m sure I’ll be back here in a year going “Why did I write it that way!?!?!?!?!?” but for now, I hope it’s simple enough for someone else to follow and possibly get inspired.

The main function is really just calling the other functions which we will break down below

Step 1 – Create the Custom View

In this code, we’re going to simply use this small function that I created to gather the name of the custom view ( the view_name variable ) and then use that to create the JSON payload ( the payload variable ). The other part of the JSON payload is the autoAddDevType variable which is hard-coded to 0. This could be used to have the system automatically add new devices of a given type, but I”m in interested in doing the automation myself here. The last part of this code will be used as the input for another function, which is the return of the view name. You can see in the main function in the the view_name = create_new_view() line.

Step 2 – Get Custom Views

For the next part, we are going to need to figure out what Id was assigned to this new view, to do this we’re going to have to go through a couple of steps. The first one is to ask the NMS to send us a list of all the known custom views. The following function will request the list, which will be returned as a JSON array, and then convert it over into a python list of dictionaries so we can work with it natively as a python object. You can see this in the main function in the view_list = get_custom_views() line

Now that we have the view_list which is the list of all the views, we’re going to have to find the ID for the new view that we just created. We do that by by using the get_view_id() function using the view_name as the input. Essentially, this will look through each of the views in the view_list that was returned above and let us know when the ‘name’ value is equal to the view_name value that we captured above. Once it’s equal, we then return the ‘symbolId’ which is the internal unique numeric value assigned to this particular custom view. This is the number we’re going to use to identify the view that that we want to add devices to. Make sense? Now that we’ve got this number, we’re going to assign it to the object view_id for use later on.

Note: I actually could have added the devices directly to the view in the original add_custom_view code code above, but then I’d have to write the modify function later if I ever wanted to change or add new devices to the view. I’m trying to follow the DRY ( Don’t Repeat Yourself ) advice here so I just write the modify here and I can then leverage it later without having to re-write the code.

Step 3 – Generate the Device List

We’re not going to spend too much time on this part as I’m essentially re-using the code from the Changing Device Categories blog. It’s pretty straight forward. I’m using some user-put to gather a list of devices that we want to add to this specific view and then capture it in the dev_list object. What’s cool about doing it this way is that the returned list will search through all the IP addresses assigned to your devices, not just the managed address which might come up in the NMS interface where you would normally perform this step. There are ways around that as well, but that’s another blog.

Step 4 – Add Devices to Existing Custom View

This last step is where things come together. We’re going to run the add_device_to_view(dev_list, view_id) function which is using the dev_list generated in step 3 and the view_id that we captured at the end of step 2 as the inputs. Essentially, we’re just saying here “ add all the devices I want to the view I just created “.

Wrapping it Up

So this is just an example of how you can tie a few pieces of code together to help automate something that might otherwise take you a lot of manual labour. In this case, just a bunch of mouse clicks and depending on the fact that you were able to manually identify all the devices you wanted to add to a specific view. Personally, I’d rather leave the hard work to the computers and move on to something that requires my brain.

Automating your NMS build – Part 3 Changing Device Categories

In the first couple of posts in this series, we created some operators, then we added some devices. In this post we’re going to look at something a little more complicated. In this post, we’re going to link a couple of different python functions together to meet the requirement which is to change a device from one category to another.

A bit about Device Categories

For those of you haven’t used HP’s Intelligent Management Centre before, the system automatically categorizes any discovered device, usually based upon SNMP sysobjectid. What this means is that when you run an auto discovery, the majority of your infrastructure will be properly classified right out of the box.

Screen Shot 2015 07 07 at 11 11 51 PM

This works great for devices which are SNMP enabled, as well as some other devices like ESX machines ( Virtual Devices ) which use SOAP as the management protocol, but it fails pretty miserably when dealing with devices which don’t support anything more than PING. IMC’s default behaviour is to put anything which doesn’t respond to SNMP in the Desktop Category.

IP Phones aren’t smart

I’ve had customers who decided to discover all of their expensive IP phones and suddenly found out that none of them were classified properly. The problem with IP phones is that they are usually pretty stupid devices. Low memory, weak CPUs. Most of the processing/thinking is done by the PBX. I’ve never seen an IP phone that supports SNMP.

In this case we have two choices

Manually change each IP phone from the Desktop category into the Voice category one device at a time.
Use the RESTful API and a bit of code to do it wihile we go have a coffee

Filtering First

So the first thing we need to do is to identify the IP phones from the rest of the devices in the Desktop category. Thankfully, this is where having a well designed network can come in REALLY handy. Most Voice networks are designed so that the IP phones are automatically put into a voice vlan. This means that all phones SHOULD be in the same layer 3 network range.

The first piece of code we need to write is a simple piece of code which will allow the user to identify which of the categories they want to filter by. Although this might sound strange, we actually need to make sure that we only grab the DESKTOP devices in a specific subnet range. Imagine if you accidentally move the router or switch for this subnet into the voice category too. Really sucks leaving when you lose your router, right?

In this piece of code, we’re simply creating a dictionary which creates the link between the categoryId, which is the number IMC internally uses to identify the defined categories and the labels we humans use to identify them. In a nutshell, this simply prints out the available categories. Why you ask? Because the human running this script needs to known which categories they want to filter by.

This next performs two different functions

Filters all known devices by the categories listed above
Filters all known devices by an IP range.

Combining the two of them we’re able to easily fine all of the devices that have been classified in the Desktop Category in the L3 subnet of the IP phones and return that as a dictionary which contains, among other things, the device IDs which we’ll need in the next step.

Putting it all together

So the last piece of code is where the magic actually happens.

First we assign the output of the filtering function above into the variable called dev_list. Essentially, this is a list of devices which meet the search criteria of the filtering function.

In our case, this means the list will consists of all the IP phones in the specific subnet that we filtered.

From there, we use the items in dev_list as input into a for loop which changes them into the new category. ( see how we use the same print_category function from above? )

Wrapping it up

Hopefully this is a fairly useful example of how putting a few basic python functions together can help to substantially cut down on the amount of time it takes to perform what’s really a simple task. That’s the whole point of automation right?

As I continue learning, I can already see there a bunch of ways to improve this code, but hopefully having a simple working example will help people who are a couple of steps behind me on this journey take another step forward.

If you’re ahead of me on this journey and have suggestions, please feel free to comment!

@netmanchris

Surfing your NMS with Python

Python is my favourite programming language. But then again, it’s also the only one I know. 🙂

I made a choice to go with python because, honestly, that’s what all the cool kids were doing at the time. But after spending the last year or so learning the basics of the language, I do find that it’s something that I can easily consume, and I’m starting to get better with all the different resources out there. BTW http://www.stackoverflow.com is your friend. you will learn to love it.

On with the show…

So in this post, I’m going to show how to use python to build a quick script that will allow you to issue the RealTimeLocate API to the HP IMC server. In theory, you can build this against any RESTful API, but I make no promises that it will work without some tinkering.

Planning the project.

I’ve written before how I’m a huge fan of OPML tools like Mind Node Pro. The first step for me was planning out the pieces I needed to make this:

usable in the future
actually work in the present

In this case I’m far more concerned about the present as I’m fairly sure that I will look back on this code in a year from now and think some words that I won’t put in print.

Aside: I’ve actually found that using the troubleshooting skills I’ve honed over the years as a network engineer helps me immensely when trying to decompose what pieces will need to go in my code. I actually think that Network Engineers have a lot of skills that are extremely transportable to the programming domain. Especially because we tend to think of the individual components and the system at the same time, not to mention our love of planning out failure domains and forcing our failures into known scenarios as much as possible.

Screen Shot 2014 11 24 at 9 33 50 PM

Auth Handler

Assuming that the RESTful service you’re trying to access will require you to authenticate, you will need an authentication handler to deal with the username/password stuff that a human being is usually required to enter. There are a few different options here. Python actually ships with URLLIB or some variant depending not the version of python you’re working with. For ease of use reasons, and because of a strong recommendation from one of my coding mentors, I chose to use the REQUESTS library. This is not shipped by default with the version of python you download over at http://www.python.org but it’s well worth the effort over PIP’ing it into your system.

The beautiful thing about REQUEST’s is that the documentation is pretty good and easily readable.

In looking through the HP IMC eAPI documentation and the Request library – I settled on the DigestAuth

Screen Shot 2014 11 24 at 10 17 07 PM

So here’s how this looks for IMC.

Building the Authentication Info

>>>import requests #imports the requests library you may need to PIP this in if you don’t have it already

>>> from requests.auth import HTTPDigestAuth    # this imports the HTTPDigestAuth method from the request library.
>>>
>>> imc_user = ”’admin”’   #The username used to auth against the HP IMC Server
>>> imc_pw = ”’admin”’   #The password of the account used to auth against the HP IMC Server.
>>>

auth = requests.auth.HTTPDigestAuth(imc_user,imc_pw) #This puts the username and password together and stores them as a variable called auth

We’ve now built the auth handler to use the username “admin” with the password “admin”. For a real environment, you’ll probably want to setup an Operator Group with only access to the eAPI functions and lock this down to a secret username and password. The eAPI is power, make sure you protect it.

Building the URL

So for this to work, I need to assign a value to the host_ip variable above so that the URL will complete with a valid response. The other thing to watch for are types. Python can be quite forgiving at times, but if you try to add to objects of the wrong type together… it mostly won’t work. So we need to make sure the host_ip is a string and the easiest way to do that is to put three quotes around the value.

In a “real” program, I would probably use the input function to allow this variable to be input as part of the flow of the program, but we’re not quite there yet.

>>> host_ip = ”’10.101.0.109”’ #variable that you can assign to a host you want to find on the network
>>> h_url = ”’http://”’ #prefix for building URLs use HTTP or HTTPS
>>> imc_server = ”’10.3.10.220:8080”’ #match port number of IMC server default 8080 or 8443
>>> url = h_url+imc_server #combines the h_url and the IP address of the IMC box as a base URL to use later
>>> find_ip_host_url = (”’/imcrs/res/access/realtimeLocate?type=2&value=”’+host_ip+”’&total=false”’) # This is the RealTimeLocate API URL with a variable set
>>>

Putting it all together.

This line takes puts the url that we’re going to send to the web server all together. You could ask “Hey man, why didn’t you just drop the whole string in one variable to begin with? “ That’s a great question. There’s a concept in programming called DRY. (Don’t Repeat Yourself). The idea is that when you write code, you should never write the same thing twice. Think in a modular fashion which would allow you to reuse pieces of code again and again.

In this example, I can easily write another f_url variable and assign to it another RESTful API that gets me something interesting from the HP IMC server. I don’t need to write the h_url portion or the server IP address portion of the header. Make sense?

>>> f_url = url + find_ip_host_url
>>> # This is a very simple mathematical operation that puts together the url and the f_url which will product the HTTP call.

Executing the code.

Now the last piece is where we actually execute the code. This will issue a get request, using the requests library. It will use the f_url as the actual URL it’s going to pass, and it will use the variable auth that we created in the Authentication Info step above to automatically populate the username and password.

The response will get returned in a variable called r.

>>> r = requests.get(f_url, auth=auth) # Using the requests library get method, we’re going to pass the f_url as the argument for the URL we’re going to access and pass auth as the auth argument to define how we authenticate Pretty simple actually .
>>>

The Results

So this is the coolest part. We can now see what’s in r. Did it work? Did we find out lost scared little host? Let’s take a look.

>>> r
<Response [200]>

Really? That’s it? .

The answer is “yes”. That’s what’s been assigned to the variable r. 200 OK may look familiar to you voice engineers who know SIP and it means mostly the same thing here. This is a response code to let you know that your request was successful – But not what we’re looking for. I want that content, right? If I do a type(r) which will tell me what python knows about what kind of object r is I will get the following.

>>> type(r)

<class ‘requests.models.Response’>

So this tells us that maybe we need to go back to the request documentation and look for info on the responses. Now we know to access the part of the response that I wanted to see, which is the reply to my request on where the host with ip address 10.101.0.111 is actually located on the network.

So let’s try out one of the options and see what we get

>>> r.content
b'<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?><list><realtimeLocation><locateIp>10.101.0.111</locateIp><deviceId>4</deviceId><deviceIp>10.10.3.5</deviceIp><ifDesc>GigabitEthernet1/0/16</ifDesc><ifIndex>16</ifIndex></realtimeLocation></list>’

How cool is that. We put in an IP address and we actually learned four new things about that IP address without touching a single GUI. And the awesome part of this? This works across any of the devices that HP IMC supports.

Where to from here?

So we’ve just started on our little journey here. Now that we have some hints to the identity of the network devices and specific interface that is currently harbouring this lost host, we need to use that data as hints to continue filling in the picture.

But that’s in the next blog…

Comments or Questions? Feel free to post below!

HP IMC’s New eAPI

Now that I got rum-pooh out of my system… on to a slightly more technical post.

Not sure if anyone of you caught the recent announcements about the new eAPI from HP’s Intelligent Management center. In a nutshell, this is a RESTful API which allows programatic access to almost all ( maybe all?) of the IMC functions through an HTTP(s) interface.

Now I’m not a programer at all, but I like to think I have a working knowledge of programing logic. At least enough to give a half-decent programmer enough information to get the job done.

So when I had a co-worker present me with a problem earlier this week, I thought “Hey, I wonder what this new eAPI can really do?”. I did mention I’m not a programer right? After this little exercise, I’m thinking I might just have to pick up some scripting skills this year. 🙂

So what’s the value of the eAPI? It functionally allows IMC to act as the progamatic upper layer APIs and abstracts the actual management task from the underlying hardware devices.

In less complicated terms; it means that a program can say “Hey, IMC change the VLAN on this port” and IMC, assuming IMC actually supports that particular device, it will change the VLAN on that port.

INDEPENDANT OF THE ACTUAL VENDOR

Yup. That’s right. IMC doesn’t care if that device is a HP 5500EI ( comware ), a HP 3800 ( procurve), or even a Cisco Catalyst 3560. From the perspective of the developer on the other side. It’s as simple as “Hey, IMC change the VLAN on this port”.

So the actual challenge I was given was the following.

” A customer wants to take a bar-code reader, scan in the MAC address of a device, plug it into the network, push a button and then have IMC automatically put it in the right VLAN”.

Now first I had to break that down into the various components.

1) Bar-code reader scans the MAC address

2) Program has to capture the MAC address for use in the %mac-address% variable in the script.

3) Find the device in the network

Hmmm… this could be more difficult than I thought.

So, I need to mock this up, so I break out to a windows CMD prompt.

Ping a known address ( my Synology NAS — LOVE THIS PRODUCT ). And then put do a arp -a to get the following output

10.101.0.51 00-11-32-10-03-8b

Now if I was using the IMC web-interface, I would just use the Resource-Terminal Access-Real-Time Location feature which will, you guessed it, locate a host in real-time using the mac-address or the IP address.

Hmmm… that kinda sucks for output for the script to leverage. So I went and looked at the eAPI documentation and came out with this little baby

The eAPI call is the following ( if you wanted to search for an IP address you would use type=2 instead of 1 )

( Don’t click this, it won’t take you anywhere)

http://10.101.0.201:8080/imcrs/res/access/realtimeLocate?type=1&value=00-11-32-10-03-8b

The return is the following

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

– <list>

– <realtimeLocation>

<locateIp>00:11:32:10:03:8b</locateIp>

<deviceIp>10.101.0.221</deviceIp>

<ifDesc>GigabitEthernet1/0/21</ifDesc>

</realtimeLocation>

</list>

Yup. Good old XML. Easy to apply transforms or grab variables. Programmers love this stuff.

4) So now I have the device IP (switch) that this is plugged into, and the ifDesc which is the actual interface it’s located on. So now I have to figure out how to apply the VLAN to this interface. So I break out the trusty eAPI documentation and start looking for the VLAN section.

Hmmm… I have the devIP and the ifDesc.. not the devID and the ifIndex

note to self: Feedback to the developers to have the first command return the devID and the ifIndex variables

So now I have to find the devID and the ifIndex for that devIP and intDesc

5a) Now if I was on the trusty IMC web interface, I would go to the device resource page… hmmm… that doesn’t appear to be there. I guess instead, let’s go to the eAPI documentation and look for something that looks like a dev query.

Yup. It’s actually called devquery. And it looks like I can filter based on the device IP.

Cool. So now I can search for the specific device IP and hope we get the devID variable back that we need for the VLAN call.

http://10.101.0.201:8080/imcrs/plat/res/device?ip=10.101.0.221

The return is the following

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

– <list>

– <device>

<id>15</id> ———————————————- This is the device ID that we need to reference later

<label>HP_5500EI</label>

<ip>10.101.0.221</ip>

<mask>255.255.255.0</mask>

<status>1</status>

<statusDesc>Normal</statusDesc>

<sysName>HP_E5500EI</sysName>

<contact>HP Montreal</contact>

<location>Marlborough, MA 01752 USA</location>

<sysOid>1.3.6.1.4.1.43.1.16.4.3.36</sysOid>

<sysDescription>HP_5500EI</sysDescription>

<devCategoryImgSrc>Switch</devCategoryImgSrc>

<topoIconName>stack</topoIconName>

<categoryId>1</categoryId>

<symbolId>1022</symbolId>

<symbolName>HP_5500EI</symbolName>

<symbolType>3</symbolType>

<symbolDesc>HP_5500EI</symbolDesc>

<symbolLevel>3</symbolLevel>

<parentId>1003</parentId>

<typeName>3Com S4800G PWR 24-Port</typeName>

<mac>00:1e:c1:dc:fc:01</mac>

<link op=”GET” rel=”self” href=”http://10.101.0.201:8080/imcrs/plat/res/device/15” />

</device>

</list>

There it is. DevID is “15”.

5b) So now we need to figure out that ifIndex value associated with <ifDesc>GigabitEthernet1/0/21</ifDesc> that we pulled above.

If I was in the webinterface, I would simply go to the device ( 10.101.0.221 ), click on the interface list, click on interface Gig 1/0/21 and I would pull out the ifIndex from the interface…

But again, those programmers don’t want HTML, they want an easy XML output that they can play with. So let’s find that…

http://10.101.0.201:8080/imcrs/plat/res/device/15/interface?start=1&size=100

This returns a whole bunch of data for all the interfaces on the switch, but I’m sure that any decent programmer can write a regex expression to only return the one who’s ifDesc value is for Gig 1/0/21, right?

– <interface>

<ifIndex>21</ifIndex> ——————————————–In this case, the ifindex value is the same as the port number. That’s not always going to be true. This is the other variable for the set VLAN

<ifType>6</ifType>

<ifTypeDesc>ETHERNETCSMACD</ifTypeDesc>

<ifDescription>GigabitEthernet1/0/21</ifDescription>

<adminStatus>1</adminStatus>

<adminStatusDesc>Up</adminStatusDesc>

<showStatus>2</showStatus>

<statusDesc>Down</statusDesc>

<operationStatus>2</operationStatus>

<operationStatusDesc>Down</operationStatusDesc>

<ifspeed>10000000</ifspeed>

<appointedSpeed>-1</appointedSpeed>

<ifAlias>GigabitEthernet1/0/15 Interface</ifAlias>

<phyAddress>00:1e:c1:dc:fc:4f</phyAddress>

<mtu>1522</mtu>

<lastChange>4 day(s) 21 hour(s) 39 minute(s) 50 second(s) 990 millisecond(s)</lastChange>

<lastChangeTime>42359099</lastChangeTime>

<filterTrapStatus>0</filterTrapStatus>

</interface>

So now we have the DevID 15 and the ifIndex for the actual interface where that MAC-address is located.

So let’s go back to the set that VLAN

Let’s assume that you wanted to put the device in VLAN 20, you would run the following

http://10.101.0.201:8080/imcrs/vlan/20?devId=15&ifIndex21

That’s about it for the task. Now any decent programmer is going to have to put in some checking and error handling, for instance, you might want to check whether or not that VLAN actually EXISTS on the switch. ( Can’t put a VLAN on a port if the VLAN doesn’t exist on the switch, right? ) or maybe return an error if the MAC-address is actually seen on two interfaces, but in a nutshell that’s it.

note: I would also suggest that the dev actually bounce the port to make sure that the device hasnt’ gotten locked in with a DHCP address on the wrong subnet.

/plat/res/device/{deviceId}/interface/{ifIndex}/down to down the interface

so for use that would be /plat/res/device/15/interface/21/down

and then immediately do a

/plat/res/device/{deviceId}/interface/{ifIndex}/up

or again for us /plat/res/device/15/21/up

Now whether the actual switch commands are “switchport acces vlan 20 ” or ” port access vlan 20 ” or some other variation on a theme doesn’t actually matter to your devOps team. They just write the code to follow the steps and IMC and the eAPI will take care of the rest.

Pretty cool stuff. 🙂

@netmanchris