Serial numbers how I love thee…

No one really like serial numbers, but keeping track of them is one of the “brushing your teeth” activities that everyone needs to take care of. It’s like eating your brussel sprouts. Or listening to your mom. You’re just better of if you do it quickly as it just gets more painful over time.

Not only is it just good hygene, but you may be subject to regulations, like eRate in the United States where you have to be able to report on the location of any device by serial number at any point in time.

Trust me, having to play hide-and-go seek with an SSH session is not something you want to do when government auditors are looking for answers.

I’m sure you’ve already guessed what I’m about to say, but I”ll say it anyway…

There’s an API for that!!!

HPE IMC base platform has a great network assets function that automatically gathers all the details of your various devices, assuming of course they supportRFC 4133, otherwise known as the Entity MIB. On the bright side, most vendors have chosen to support this standards based MIB, so chances are you’re in good shape.

And if they don’t support it, they really should. You should ask them. Ok?

So without further ado, let’s get started.

 

Importing the required libraries

I’m sure you’re getting used to this part, but it’s import to know where to look for these different functions. In this case, we’re going to look at a new library that is specifically designed to deal with network assets, including serial numbers.

In [1]:
from pyhpeimc.auth import *
from pyhpeimc.plat.netassets import *
import csv
In [2]:
auth = IMCAuth("http://", "10.101.0.203", "8080", "admin", "admin")
In [3]:
ciscorouter = get_dev_asset_details('10.101.0.1', auth.creds, auth.url)
 

How many assets in a Cisco Router?

As some of you may have heard, HPE IMC is a multi-vendor tool and offers support for many of the common devices you’ll see in your daily travels.

In this example, we’re going to use a Cisco 2811 router to showcase the basic function.

Routers, like chassis switches have multiple components. As any one who’s ever been the victem owner of a Smartnet contract, you’ll know that you have individual components which have serial numbers as well and all of them have to be reported for them to be covered. So let’s see if we managed to grab all of those by first checking out how many individual items we got back in the asset list for this cisco router.

In [4]:
len(ciscorouter)
Out[4]:
7
 

What’s in the box???

Now we know that we’ve got an idea of how many assets are in here, let’s take a look to see exactly what’s in one of the asset records to see if there’s anything useful in here.

In [5]:
ciscorouter[0]
Out[5]:
{'alias': '',
 'asset': 'http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=1',
 'assetNumber': '',
 'boardNum': 'FHK1119F1DX',
 'bom': '',
 'buildInfo': '',
 'cleiCode': '',
 'containedIn': '0',
 'desc': '2811 chassis',
 'devId': '15',
 'deviceIp': '10.101.0.1',
 'deviceName': 'router.lab.local',
 'firmwareVersion': 'System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)',
 'hardVersion': 'V04 ',
 'isFRU': '2',
 'mfgName': 'Cisco',
 'model': 'CISCO2811',
 'name': '2811 chassis',
 'phyClass': '3',
 'phyIndex': '1',
 'physicalFlag': '0',
 'relPos': '-1',
 'remark': '',
 'serialNum': 'FHK1119F1DX',
 'serverDate': '2016-01-26T15:20:40-05:00',
 'softVersion': '15.1(4)M, RELEASE SOFTWARE (fc1)',
 'vendorType': '1.3.6.1.4.1.9.12.3.1.3.436'}
 

What can we do with this?

With some basic python string manipulation we could easily print out some of the attributes that we want into what could easily turn into a nicely formated report.

Again realise that the example below is just a subset of what’s available in the JSON above. If you want more, just add it to the list.

In [7]:
for i in ciscorouter:
    print ("Device Name: " + i['deviceName'] + " Device Model: " + i['model'] +
           "\nAsset Name is: " + i['name'] + " Asset Serial Number is: " +
           i['serialNum']+ "\n")
 
Device Name: router.lab.local Device Model: CISCO2811
Asset Name is: 2811 chassis Asset Serial Number is: FHK1119F1DX

Device Name: router.lab.local Device Model: VIC2-2FXO
Asset Name is: 2nd generation two port FXO voice interface daughtercard on Slot 0 SubSlot 2 Asset Serial Number is: FOC11063NZ4

Device Name: router.lab.local Device Model:
Asset Name is: 40GB IDE Disc Daughter Card on Slot 1 SubSlot 0 Asset Serial Number is: FOC11163P04

Device Name: router.lab.local Device Model:
Asset Name is: AIM Container Slot 0 Asset Serial Number is:

Device Name: router.lab.local Device Model:
Asset Name is: AIM Container Slot 1 Asset Serial Number is:

Device Name: router.lab.local Device Model:
Asset Name is: C2811 Chassis Slot 0 Asset Serial Number is:

Device Name: router.lab.local Device Model:
Asset Name is: C2811 Chassis Slot 1 Asset Serial Number is:

 

Why not just write that to disk?

Although we could go directly to the formated report without a lot of extra work, we would be losing a lot of data which we may have use for later. Instead why don’t we export all the available data from the JSON above into a CSV file which can be later opened in your favourite spreadsheet viewer and manipulated to your hearst content.

Pretty cool, no?

In [9]:
keys = ciscorouter[0].keys()
with open('ciscorouter.csv', 'w') as file:
    dict_writer = csv.DictWriter(file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(ciscorouter)
 

Reading it back

Now we’ll read it back from disk to make sure it worked properly. When working with data like this, I find it useful to think about who’s going to be consuming the data. For example, when looking at this remember this is a CSV file which can be easily opened in python, or something like Microsoft Excel to manipuate further. It’s not realy intended to be read by human beings in this particular format. You’ll need another program to consume and munge the data first to turn it into something human consumable.

In [12]:
with open('ciscorouter.csv') as file:
    print (file.read())
 
firmwareVersion,vendorType,phyIndex,relPos,boardNum,phyClass,softVersion,serverDate,isFRU,alias,bom,physicalFlag,deviceName,deviceIp,containedIn,cleiCode,mfgName,desc,name,hardVersion,remark,asset,model,assetNumber,serialNum,buildInfo,devId
"System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)",1.3.6.1.4.1.9.12.3.1.3.436,1,-1,FHK1119F1DX,3,"15.1(4)M, RELEASE SOFTWARE (fc1)",2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,0,,Cisco,2811 chassis,2811 chassis,V04 ,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=1,CISCO2811,,FHK1119F1DX,,15
,1.3.6.1.4.1.9.12.3.1.9.3.114,14,0,FOC11063NZ4,9,,2016-01-26T15:20:40-05:00,1,,,2,router.lab.local,10.101.0.1,13,,Cisco,2nd generation two port FXO voice interface daughtercard,2nd generation two port FXO voice interface daughtercard on Slot 0 SubSlot 2,V01 ,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=14,VIC2-2FXO,,FOC11063NZ4,,15
,1.3.6.1.4.1.9.12.3.1.9.15.25,30,0,FOC11163P04,9,,2016-01-26T15:20:40-05:00,1,,,2,router.lab.local,10.101.0.1,29,,Cisco,40GB IDE Disc Daughter Card,40GB IDE Disc Daughter Card on Slot 1 SubSlot 0,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=30, ,,FOC11163P04,,15
,1.3.6.1.4.1.9.12.3.1.5.2,25,6,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,3,,Cisco,AIM Container Slot 0,AIM Container Slot 0,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=25,,,,,15
,1.3.6.1.4.1.9.12.3.1.5.2,26,7,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,3,,Cisco,AIM Container Slot 1,AIM Container Slot 1,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=26,,,,,15
,1.3.6.1.4.1.9.12.3.1.5.1,2,0,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,1,,Cisco,C2811 Chassis Slot,C2811 Chassis Slot 0,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=2,,,,,15
,1.3.6.1.4.1.9.12.3.1.5.1,27,1,,5,,2016-01-26T15:20:40-05:00,2,,,0,router.lab.local,10.101.0.1,1,,Cisco,C2811 Chassis Slot,C2811 Chassis Slot 1,,,http://10.101.0.203:8080/imcrs/netasset/asset/detail?devId=15&phyIndex=27,,,,,15

 

What about all my serial numbers at once?

That’s a great question! I’m glad you asked. One of the most beautiful things about learning to automate things like asset gathering through an API is that it’s often not much more work to do something 1000 times than it is to do it a single time.

This time instead of using the get_dev_asset_details function that we used above which gets us all the assets associated with a single device, let’s grab ALL the devices at once.

In [13]:
all_assets = get_dev_asset_details_all(auth.creds, auth.url)
In [14]:
len (all_assets)
Out[14]:
1013
 

That’s a lot of assets!

Exactly why we automate things. Now let’s write the all_assets list to disk as well.

**note for reasons unknown to me at this time, although the majority of the assets have 27 differnet fields, a few of them actually have 28 different attributes. Something I’ll have to dig into later.

In [15]:
keys = all_assets[0].keys()
with open('all_assets.csv', 'w') as file:
    dict_writer = csv.DictWriter(file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(all_assets)
 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-e4c553049911> in <module>()
 3     dict_writer = csv.DictWriter(file, keys)
 4     dict_writer.writeheader()
----> 5dict_writer.writerows(all_assets)

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/csv.py in writerows(self, rowdicts)
 156         rows = []
 157         for rowdict in rowdicts:
--> 158rows.append(self._dict_to_list(rowdict))
 159         return self.writer.writerows(rows)
 160

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/csv.py in _dict_to_list(self, rowdict)
 147             if wrong_fields:
 148                 raise ValueError("dict contains fields not in fieldnames: "
--> 149 + ", ".join([repr(x) for x in wrong_fields]))  150         return [rowdict.get(key, self.restval) for key in self.fieldnames]
 151

ValueError: dict contains fields not in fieldnames: 'beginDate'
 

Well That’s not good….

So it looks like there are a few network assets that have a different number of attributes than the first one in the list. We’ll write some quick code to figure out how big of a problem this is.

In [16]:
print ("The length of the first items keys is " + str(len(keys)))
for i in all_assets:
    if len(i) != len(all_assets[0].keys()):
       print ("The length of index " + str(all_assets.index(i)) + " is " + str(len(i.keys())))
 
The length of the first items keys is 27
The length of index 39 is 28
The length of index 41 is 28
The length of index 42 is 28
The length of index 474 is 28
The length of index 497 is 28
The length of index 569 is 28
The length of index 570 is 28
The length of index 585 is 28
The length of index 604 is 28
The length of index 605 is 28
The length of index 879 is 28
The length of index 880 is 28
The length of index 881 is 28
The length of index 882 is 28
The length of index 883 is 28
The length of index 884 is 28
The length of index 885 is 28
The length of index 886 is 28
 

Well that’s not so bad

It looks like the items which don’t have exactly 27 attribues have exactly 28 attributes. So we’ll just pick one of the longer ones to use as the headers for our CSV file and then run the script again.

For this one, I’m going to ask you to trust me that the file is on disk and save us all the trouble of having to print out 1013 seperate assets into this blog post.

In [18]:
keys = all_assets[879].keys()
with open ('all_assets.csv', 'w') as file:
    dict_writer = csv.DictWriter(file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(all_assets)
 

What’s next?

So now that we’ve got all of our assets into a CSV file which is easily consumable by something like Excel, you can now chose what to do with the data.

For me it’s interesting to see how vendors internally instrument their boxes. Some have serial numbers on power supplies and fans, some don’t. Some use the standard way of doing things. Some don’t.

From an operations perspective, not all gear is created equal and it’s nice to understand what’s supported when trying to make a purchasing choice for something you’re going to have to live with for the next few years.

If you’re looking at your annual SMARTnet upgrade, at least you’ve now got a way to easily audit all of your discovered environment and figure out what line cards need to be tied to a particualr contract.

Or you could just look at another vendor who makes your life easier. Entirely your choice.

@netmanchris

Advertisements

Automating your NMS build using Python and Restful APIs Part 2 – Adding Devices though Auto-Discovery

This is the second in a series of posts where I’ll be using a RESTful API to automate a bunch of the initial deployment functions within my NMS. There are a bunch of reasons to do this that are right for the business. Being able to push information gathering onto the customer, being able to use lower-skilled ( and hence lower paid!) resources to do higher level tasks. Being able to be more efficient in your delivery, undercut the competitors on price and over deliver on quality. It’s a really good project to sink my teeth and use some of my growing coding skills to make a difference to the business. 

Other posts in this series

Automating your NMS build using Python and Restful APIs Part 1 – Creating Operators

 

Adding Devices

A Network Management System without devices is a sad, sad thing. It’s just a lonely piece of code with no purpose in life. Empty graphs and arm notifications with nothing to notify. Today we’re going to look at adding devices through launching a basic network range auto discovery. In a follow up post, I’ll be adding device through a CSV import which is my preferred method. 

One thing which I will not be covering in this post to save some time and space is how to authenticate to the NMS system which is covered here

Launching an Autodiscovery

One of the worst things about installing an NMS in a customers environment is the inevitable scope creep that happens. The major reason that this happens is that most customers looking for an NMS are doing so because they really don’t have a clue what’s in their network, and potentially not even how they are configured. They know it’s kinda working most of the time, but not much more than that.

Typically, you have a statement of work to discover N number of devices. The customer says  “I don’t know all the IP addresses, why don’t you just discover this range of IP addresses. So you launch a discovery from your tool and suddenly, you’ve got 2*N devices in your database!  And because they are in your database, you’re now responsible for getting them up and running. Even though your contract is for only N devices, you don’t want to disappoint your customer right? 

There other things that can easily go wrong in an auto discovery as well like

  • Mis-matched SNMP strings: You have the SNMP read, but not the write strings.

This can cause some unpredictable results that are SOOO fun to troubleshoot.  

  • Missing CLI credentials: You have SNMP strings, but not the CLI strings.

There are some things that are only accessible through CLI. It’s great that we’re moving into an era where decent programatic APIs are available devices, but for the mass majority of your infrastructure, there’s a ton of functions which are only available through the good old command line interface.  If you don’t have the right credentials, you’re going to have problems. In my experience, unless they have a centralized authentication system in place, like Cisco ACS, FreeRADIUS or other; you’re probably going to have to troubleshoot half of the device credentials. 

Let’s look at the code. 

First I’ll show the entire function complete, and then we’ll break down the sections individually.  Again, I’ll be skipping the imc_creds() function as it’s covered in the earlier post. 

 import requests, json, sys, time, subprocess, csv, os, ipaddress 

 def plat_auto_discover():

    if auth == None or url == None: # checks to see if the imc credentials are already available
    imc_creds()
    auto_discover_url = '/imcrs/plat/res/autodiscover/start'
    f_url = url + auto_discover_url
    network_address = input(
    '''What is the the network address of the range you wish to discover?\nPlease input the address in the format "192.168.0.0/24": ''')
    # end_address = input('''What is the last address of the network range you wish to discover?\nIPv4 Address: ''')
    try:
        network_address = ipaddress.ip_network(network_address)
    except ValueError:
        print("You have entered an invalid network address. Please try again.")
        time.sleep(2)
        print ('\n'*80)
        plat_auto_discover()
    payload = ''' {
    "mode": "0",
    "ipSection": {
        "begin": "''' + str(network_address[1]) + '''",
        "end": "''' + str(network_address[-2]) + '''"
        },
    "discoverNonSnmpDevice": "true",
    "pingAll": "true"
    }
    '''
    r = requests.post(f_url, data=payload, auth=auth, headers=headers) #creates the URL using the payload variable as the contents
    if r.status_code == 200:
        print ("Auto-Discovery Successfully Started")
    else:
        print ("An Error has occured")

 Gathering the network range

So the first task here is gathering the actual network range that we want to run the auto discover over. For this example, we’ll keep this very basic and use a simple 192.168.0.0/24 network. This is going to cause the system to scan the entire 254 hosts of the subnet.  One of the other things to keep in mind here is that I’m going to use the default system templates for SNMP and Telnet which is why I’m not gathering them here. This isn’t magic, right?

So I’m using the input function to gather the IP address network range that I want to discover. I’m also using the python ipaddress standard library to caste the user input as a network_address to ensure that it’s a valid input to the final function. 

network_address = input(
    ”’What is the the network address of the range you wish to discover?\nPlease input the address in the format “192.168.0.0/24″: ”’)
    # end_address = input(”’What is the last address of the network range you wish to discover?\nIPv4 Address: ”’)
    try:
        network_address = ipaddress.ip_network(network_address)
    except ValueError:
        print(“You have entered an invalid network address. Please try again.”)
        time.sleep(2)
        print (‘\n’*80)
        plat_auto_discover()

So essentially, this code performs three steps

  1. Gathers the desired network address discovery range in the 192.168.0.0/24 format
  2. Attempts to use the ipaddress.ip_network method from the python ipaddress library to test if the user input was valid, and stores it in the  variable network_address
  3. If the ipaddress.ip_network method fails, this will raise an error and re-run the function to gather input in the right format.

The other nice thing about storing the user input as a ipaddress.ip_network object means that we can easily gather the start and end of this subnet based on the subnet math that’s included in the library. I’d always prefer  to have someone/something else do the binary math. 🙂

Creating the JSON Array

This particular API uses a JSON array in the HTTP message body to gather all of the information used to launch the auto-discovery.

payload = ”’

{
    “mode”: “0”,
    “ipSection”: {
        “begin”: “”’ + str(network_address[1]) + ”'”,
        “end”: “”’ + str(network_address[-2]) + ”'”
        },
    “discoverNonSnmpDevice”: “true”,
    “pingAll”: “true”
}
”’

I’m assuming you’re somewhat comfortable with working with strings in python here. The only thing that’s a little funny is the two str(network_address[1]) and str(network_address[-2]) lines.  This is the magic part of the ipaddress library that I mentioned above. These two lines of code do all the binary math for you. For those who aren’t used to subnet math, the first address in any IP network range is actually the network address, so we don’t want the 1, not 0, will be the first valid IP address in the range.  The last address in the range is actually the broadcast address, which is why we use -1 and not -2.  Pretty obvious when you think about it, right?

The network_address object is actually of the ‘ipaddress.IPv4Network’ class, so I’m using the str method to ensure that it’s a valid string when I’m adding joining it to the other text to create the payload. 

You can see from the following print statement that the first and last address in the range are exactly what we expect them to be. 

>>> print (json.dumps(json.loads(payload),indent=4))
{
"pingAll": "true",
"mode": "0",
"discoverNonSnmpDevice": "true",
"ipSection": {
"begin": "192.168.0.1",
"end": "192.168.0.254"
}
}

 

Sending the Request

So the last part of this code is just sending the actual request to the web server and seeing what happens.

 

r = requests.post(f_url, data=payload, auth=auth, headers=headers) #creates the URL using the payload variable as the contents 
    if r.status_code == 200:
        print ("Auto-Discovery Successfully Started")
    else:
        print ("An Error has occured")

 
Using the requests library POST method, we create the HTTP call and include the PAYLOAD that we created above as the data for the message body of this request. 
I’m also evaluating the return of the request here. If the HTTP response code is 200 OK, then everything is good. If it’s anything else, then there’s an issue. As with almost any code on the planet, we could probably do a lot more error handling here, but more my purposes, this is more than ok.
 
 
Do you have better ways of doing anything I’ve got here?  Comments are welcome
 

Automating your NMS build using Python and Restful APIs Part 1 – Creating Operators

It’s a funny world we live in.  Unless you’re hiding under a rock, there’s been a substantial push in the industry over the last few years to move away from the CLI.  As someone right in the middle of this swirling vortex of inefficiency, I’d like to suggest that it’s not so much the CLI that’s the problem, but the fact that each box is handled on an individual basis and that human beings access the API through a keyboard. Not exactly next-generation technology.

 

I’ve been spending lot of time learning python and trying to apply it to my daily tasks. I started looking at the HP IMC Network Management station a few months ago. Mainly as a way to start learning about how I can use python to access RESTFul APIs as well as gain some hands on working with JSON and XML. As an observation, it’s interesting to be that I’m using a CLI ( python ) to configure an NMS ( IMC) that I’m using to avoid using the CLI. ( network devices ).   

I’ve got a project I’m working on to try and automate a bunch of the initial deployment functions within my NMS. There are a bunch of reasons to do this that are right for the business. Being able to push information gathering onto the customer, being able to use lower-skilled ( and hence lower paid!) resources to do higher level tasks. Being able to be more efficient in your delivery, undercut the competitors on price and over deliver on quality. It’s a really good project to sink my teeth and use some of my growing coding skills to make a difference to the business. 

This is the first post in which I’ll discuss and document some of the simple functions I’m developing. I make no claims to be a programmer, or even a coder. But I’m hoping someone can find something here usefull, and possibly get inspired to start sharing whatever small project you’re working on as well. 

 

Without further ado, let’s jump in and look at some code. 

What’s an Operator

Not familiar with HP IMC?  You should be! It’s chock full of goodness and you can get a 60 day free trial here.   In IMC an Operator is someone who has the right to log into the system and perform tasks in the NMS itself.  The reason they use the word operator vs. user is that there’s a full integrated BYOD solution available as an add-on module which treats a user as resource, which of course is not the same thing as an administrator on the system. 

IMC’s got a full RBAC system as well which allows you to assign different privilege levels to your operators, from view only to root-equiv access, as well as splitting up what devices you can perform actions on, as well as segmenting what actions you’re allowed to perform. Pretty powerful stuff once you understand how the pieces go together. 

Adding an Operator in the GUI

 This is a screen capture of the dialog used to add an operator into IMC.  It’s intuitive. You put the username in the username box, you put the password in the password box. Pretty easy right?

If you know what you’re doing and you’re a reasonably good typist, you can add probably add an operator in a minute or less.  

Screen Shot 2015 04 16 at 12 19 17 PM

Where do Operators come from?

Don’t worry. This isn’t a birds and bees conversation.  One of the biggest mistakes that I see when people start into any network management system project, whether that’s Solarwinds, Cisco Prime, What’s up Gold, HP NNMi, or HP IMC, is that they don’t stop to think about what they want/need to do before they start the project.  They typically sit down, start an auto-discovery and then start cleaning up afterwards.  Not exactly the best way to ensure success in your project is it?

When I get involved in a deployment project, I try to make sure I do as much of the information gathering up front. This means I have a bunch of excel spreadsheets that I ask them to fill in before I even arrive onsite. This ensures two things:

  1. I can deliver on what the customer actually wants
  2.  I know when I’m done the project and get to walk away and submit the invoice. 

 

I won’t make any judgement call on which one of those is more important. 

 

 

My Operator Template

My operator template looks like this

NewImage

The values map to the screen shot above exactly as you would expect them to. 

Full name is the full name. Name is the login name, password is the password etc…  

The authType is a little less intuitive, although it is documented in the API docs. The authType maps to the authentication type above which allows you to choose how this specific operator is going to authenticate, through local auth, LDAP, or RADIUS. 

The operator group, which is “1” in my example, maps to the admin operator group which means that I have root-level access on the NMS and can do anything I want. Which is, of course, how it should be, right?

 

The Problem

So I’ve got a CSV file and I know it takes about one minute to create an operator because I can type and I know the system. Why am I automating this? Well, there are a couple of reasons for that.

  • Because I can and I want to gain more python experience
  • Because if I have to add ten operators, this just became ten minutes.
  • Because I already have the CSV file from the customer. Why would I type all this stuff again?
  • Because I can reuse this same format at every customer project I get involved in. 
  • Because I can blame any typos on the customer

Given time, I could add to this list, but let’s just get to the code. 

The Code

Authenticating to the Restful API

Although the auth examples in the eAPI documentation use the standard URLIB HTTP library, I’ve found that the requests library is MUCH more user friendly and easier to work with.

So I first create a couple of global variables called URL and AUTH that I will use to store the credentials.  

 

#url header to preprend on all IMC eAPI calls
url = None

#auth handler for eAPI calls
auth = None 

Now we get to the meat. I think this is pretty obvious, but this function gathers the username and password used to access the eAPI and then tests it out to make sure it’s valid. Once it’s verified as working ( The 200 OK check ). The credentials are then stored in the URL and AUTH global variables for use later on. I’m sure someone could argue that I shouldn’t be using global variables here, but it works for me. :) 
 
def imc_creds():
    ''' This function prompts user for IMC server information and credentuials and stores
    values in url and auth global variables'''
    global url, auth, r
    imc_protocol = input("What protocol would you like to use to connect to the IMC server: \n Press 1 for HTTP: \n Press 2 for HTTPS:")
    if imc_protocol == "1":
        h_url = 'http://'
    else:
        h_url = 'https://'
    imc_server = input("What is the ip address of the IMC server?")
    imc_port = input("What is the port number of the IMC server?")
    imc_user = input("What is the username of the IMC eAPI user?")
    imc_pw = input('''What is the password of the IMC eAPI user?''')
    url = h_url+imc_server+":"+imc_port
    auth = requests.auth.HTTPDigestAuth(imc_user,imc_pw)
    test_url = '/imcrs'
    f_url = url+test_url
    try:
        r = requests.get(f_url, auth=auth, headers=headers)
    except requests.exceptions.RequestException as e: #checks for reqeusts exceptions
        print ("Error:\n"+str(e))
        print ("\n\nThe IMC server address is invalid. Please try again\n\n")
        imc_creds()
    if r.status_code != 200: #checks for valid IMC credentials
        print ("Error: \n You're credentials are invalid. Please try again\n\n")
        imc_creds()
    else:
        print ("You've successfully access the IMC eAPI")
 
 
I”m using this function to gather the credentials of the operator accessing the API. By default when you first install HP IMC, these are admin/admin.    You could ask: Why don’t you just hardcode those into the script? Why bother with writing a function for this? 
Answer: Because I want to reuse this as much as possible and there are lots of things that you can do with the eAPI that you would NOT want just anyone doing. Plus, hardcoding the username and password of the NSM system that controls your entire network is just a bad idea in my books. 
 

Creating the Operators

I used the HP IMC eAPI /plat/operator POST call to as the basis for this call. 

Screen Shot 2015 04 16 at 1 06 21 PM

 

After doing a bit of testing, I arrived at a JSON array which would allow me to create an operator using the “Try it now” button in the API docs.  ( http://IMC_SERVER:PORTNUMBER/imcrs to access the online docs BTW ).

    {
"password": "access4chris",
"fullName": "Christopher Young",
"defaultAcl": "0",
"operatorGroupId": "1",
"name": "cyoung",
"authType": "0",
"sessionTimeout": "10",
"desc": "admin account"
}

Using the Try it now button, you can also see the exact URL that is used to call this API. 

The 201 response below means that it was successfully executed. ( you might want to read up on HTTP codes as it’s not quite THAT simple, but for our purposes, it will work ).

Screen Shot 2015 04 16 at 1 10 46 PM

Now that I’ve got a working JSON array and the URL I need, I’ve got all the pieces I need to put this small function together. 

You can see the first thing I do is check to see if the auth and url variables are still set to None. If they are still None I use the IMC_CREDS function from above to gather them and store them. 

 

I create another variables called headers which stores the headers for the HTTP call. By default, the HP IMC eAPI will respond with XML. After working with XML for a few months, I decided that I prefer JSON. It just seems easier for me to work with.

This piece of code takes the CSV file that we created above and decodes the CSV file into a python dictionary using the column headers as the key and any additional rows as the values. This is really cool in that I can have ten rows, 50 rows, or 100 rows and it doesn’t matter. This script will handle any reasonable number you throw at it. ( I’ve tested up to 20 ).

 

#headers forcing IMC to respond with JSON content. XML content return is the default

headers = {‘Accept’: ‘application/json’, ‘Content-Type’: ‘application/json’,’Accept-encoding’: ‘application/json’}

def create_operator():
    if auth == None or url == None: #checks to see if the imc credentials are already available
        imc_creds()
    create_operator_url = ‘/imcrs/plat/operator’
    f_url = url+create_operator_url
    with open (‘imc_operator_list.csv’) as csvfile: #opens imc_operator_list.csv file
        reader = csv.DictReader(csvfile) #decodes file as csv as a python dictionary
        for operator in reader:
            payload = json.dumps(operator, indent=4) #loads each row of the CSV as a JSON string
            r = requests.post(f_url, data=payload, auth=auth, headers=headers) #creates the URL using the payload variable as the contents
            if r.status_code == 409:
                print (“Operator Already Exists”)
            elif r.status_code == 201:
                print (“Operator Successfully Created”)

 Now you run this code and you’ve suddenly got all the operators in the CSV file imported into your system. 

Doing some non-scientific testing, meaning I counted in Mississippi’s, it took me about 3 seconds to create 10 operators using this method.  

Time isn’t Money

Contrary to the old saying, time isn’t actually money. We can always get more money. There’s lots of ways to do that. Time on the other hand can never be regained. It’s a finite resource and I’d like to spend as much of it as I can on things that I enjoy.  Creating Operators in an NMS doesn’t qualify.

Now, I hand off a CSV file to the customer, make them fill out all the usernames and passwords and then just run the script. they have all the responsibility for the content and all I have to do is a visual on the CSV file to make sure that they didn’t screw anything up.

 

Questions or comments or better ways to do this?  Feel free to post below. I’m always looking to learn.

 

@netmanchris 

 

Surfing your NMS with Python

Python is my favourite programming language. But then again, it’s also the only one I know. 🙂

I made a choice to go with python because, honestly, that’s what all the cool kids were doing at the time. But after spending the last year or so learning the basics of the language, I do find that it’s something that I can easily consume, and I’m starting to get better with all the different resources out there. BTW http://www.stackoverflow.com is your friend.  you will learn to love it.

On with the show…

So in this post, I’m going to show how to use python to build a quick script that will allow you to issue the RealTimeLocate API to the HP IMC server. In theory, you can build this against any RESTful API, but I make no promises that it will work without some tinkering.

Planning the project.

I’ve written before how I’m a huge fan of OPML tools like Mind Node Pro.  The first step for me was planning out the pieces I needed to make this:

  • usable in the future
  • actually work in the present
In this case I’m far more concerned about the present as I’m fairly sure that I will look back on this code in a year from now and think some words that I won’t put in print.
Aside: I’ve actually found that using the troubleshooting skills I’ve honed over the years as a network engineer helps me immensely when trying to decompose what pieces will need to go in my code. I actually think that Network Engineers have a lot of skills that are extremely transportable to the programming domain. Especially because we tend to think of the individual components and the system at the same time, not to mention our love of planning out failure domains and forcing our failures into known scenarios as much as possible.

Screen Shot 2014 11 24 at 9 33 50 PM

Auth Handler

Assuming that the RESTful service you’re trying to access will require you to authenticate, you will need an authentication handler to deal with the username/password stuff that a human being is usually required to enter. There are a few different options here. Python actually ships with URLLIB or some variant depending not the version of python you’re working with.  For ease of use reasons, and because of a strong recommendation from one of my coding mentors, I chose to use the REQUESTS library.  This is not shipped by default with the version of python you download over at http://www.python.org but it’s well worth the effort over PIP’ing it into your system.

The beautiful thing about REQUEST’s is that the documentation is pretty good and easily readable.

In looking through the HP IMC eAPI documentation and the Request library – I settled on the DigestAuth

Screen Shot 2014 11 24 at 10 17 07 PM

So here’s how this looks for IMC.

Building the Authentication Info

>>>import requests   #imports the requests library you may need to PIP this in if you don’t have it already

>>> from requests.auth import HTTPDigestAuth    # this imports the HTTPDigestAuth method from the request library.
>>>
>>> imc_user = ”’admin”’   #The username used to auth against the HP IMC Server
>>> imc_pw = ”’admin”’   #The password of the account used to auth against the HP IMC Server.
>>>  

auth = requests.auth.HTTPDigestAuth(imc_user,imc_pw)     #This puts the username and password together and stores them as a variable called auth

We’ve now built the auth handler to use the username “admin” with the password “admin”. For a real environment, you’ll probably want to setup an Operator Group with only access to the eAPI functions and lock this down to a secret username and password. The eAPI is power, make sure you protect it.

Building the URL

So for this to work, I need to assign a value to the host_ip  variable above so that the URL will complete with a valid response. The other thing to watch for are types. Python can be quite forgiving at times, but if you try to add to objects of the wrong type together… it mostly won’t work.  So we need to make sure the host_ip is a string and the easiest way to do that is to put three quotes around the value.

In a “real” program, I would probably use the input function to allow this variable to be input as part of the flow of the program, but we’re not quite there yet.

>>> host_ip = ”’10.101.0.109”’   #variable that you can assign to a host you want to find on the network
>>> h_url = ”’http://”&#8217;    #prefix for building URLs use HTTP or HTTPS
>>> imc_server = ”’10.3.10.220:8080”’   #match port number of IMC server default 8080 or 8443
>>> url = h_url+imc_server    #combines the h_url and the IP address of the IMC box as a base URL to use later
>>> find_ip_host_url = (”’/imcrs/res/access/realtimeLocate?type=2&value=”’+host_ip+”’&total=false”’)   # This is the RealTimeLocate API URL with a variable set
>>>

Putting it all together.

This line takes puts the url that we’re going to send to the web server all together. You could ask “Hey man, why didn’t you just drop the whole string in one variable to begin with? “   That’s a great question.  There’s a concept in programming called DRY. (Don’t Repeat Yourself).  The idea is that when you write code, you should never write the same thing twice. Think in a modular fashion which would allow you to reuse pieces of code again and again.

In this example, I can easily write another f_url variable and assign to it another RESTful API that gets me something interesting from the HP IMC server. I don’t need to write the h_url portion or the server IP address portion of the header.  Make sense?

>>> f_url = url + find_ip_host_url
>>>    #  This is a very simple mathematical operation that puts together the url and the f_url which will product the HTTP call. 

Executing the code.

Now the last piece is where we actually execute the code. This will issue a get request, using the requests library.  It will use the f_url as the actual URL it’s going to pass, and it will use the variable auth that we created in the Authentication Info step above to automatically populate the username and password.

The response will get returned in a variable called r.

>>> r = requests.get(f_url, auth=auth)    #  Using the requests library get method, we’re going to pass the f_url as the argument for the URL we’re going to access and pass auth as the auth argument to define how we authenticate Pretty simple actually . 
>>>

The Results

So this is the coolest part. We can now see what’s in r.  Did it work? Did we find out lost scared little host?  Let’s take a look.

>>> r
<Response [200]>

Really? That’s it? .

The answer is “yes”.  That’s what’s been assigned to the variable r.  200 OK may look familiar to you voice engineers who know SIP and it means mostly the same thing here. This is a response code to let you know that your request was successful – But not what we’re looking for. I want that content, right?  If I do a type(r) which will tell me what python knows about what kind of object r is I will get the following.

>>> type(r)

<class ‘requests.models.Response’>

So this tells us that maybe we need to go back to the request documentation and look for info on the responses. Now we know to access the part of the response that I wanted to see, which is the reply to my request on where the host with ip address 10.101.0.111 is actually located on the network.

So let’s try out one of the options and see what we get

>>> r.content
b'<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?><list><realtimeLocation><locateIp>10.101.0.111</locateIp><deviceId>4</deviceId><deviceIp>10.10.3.5</deviceIp><ifDesc>GigabitEthernet1/0/16</ifDesc><ifIndex>16</ifIndex></realtimeLocation></list>’

How cool is that. We put in an IP address and we actually learned four new things about that IP address without touching a single GUI. And the awesome part of this?  This works across any of the devices that HP IMC supports.

Where to from here?

So we’ve just started on our little journey here.  Now that we have some hints to the identity of the network devices and specific interface that is currently harbouring this lost host, we need to use that data as hints to continue filling in the picture.

But that’s in the next blog…

Comments or Questions?  Feel free to post below!

Another eAPI post – What can you do with a QR tag?

So we’ve had some ambitious little engineers with too much caffeine and some new toys to play with.

For those of you who are interested, I’m going to post the source over to

which is the new IMC forum site that @neelixx setup. This code was not created by a professional code, but at least it’s a proof of concept of what you can do with a little bit of time and some knowledge.

P.S.  If you don’t have a QR code reader on your phone yet, and you didn’t just click on the image…. that’s   www.netopscommunity.net

@netmanchris

A Network Services Platform

So things are starting to get interesting with the HP IMC eAPI that was recently released. It’s really amazing to see the types of creative projects when technical people are presented with new toys. 🙂

So for those of you who didn’t read my last eAPI blog post, let me catch you up. The eAPI is a RESTFul inteface that allows programmers, or scripters to leverage the various network services that HP IMC presents.

Thanks@ioshintsfor a quick look at SNMP vs. RESTfull interface

Basically it looks a little like this.

note: This is not a full list of the IMC modules or services. Check out the HP website for a complete list.

The RESTfull inteface presents the services in a XML format which is consumable to any programming language that can parse XML. ( I’m not a programmer, but that’s pretty much all of the current ones from what I understand ).

Those services are then applied to specific devices. But what’s COOL about this, is the following.

Say you want to change a VLAN on a bunch of ports. Some of those happen to be HP Comware switches, some of them happen to be HP Procurve Switches, and some of them happen to be Cisco switches. The IMC device adapters at the bottom do all the work for you, providing a device abstraction layer so that you can just say ” add VLAN” rather than having to worry about the syntax of all the individual devices.

So what’s actually available in the HP IMC eAPI? Well you can checkout @neelixx’s blog for the documentation. This is the first release, but I’m told that the eAPI will continue to grow with each future release of the platform AND the modules.

But I think what’s a LOT more interesting is some of the projects that have started to creep up.

For example

1) Wouldn’t it be cool if when you sent someone an outlook invite for a meeting in your office that your network access control system would automatically create guest accounts for the day of the meeting and send them to your guests?

2) Wouldn’t it be cool if when your support desk could simply click on a user in Microsoft Lync and automatically see where they have been logged in the network? Check out what access service is assigned to them. Maybe they are having trouble accessing some resources and you want to make sure they are in the right VLAN.

I’ve also started to see other apps pop up such as an application that searches the entire network for the mac addresses of lost laptops and locates the interface they are plugged into. Pretty handy for a hospital where a lost laptop with patient data is a nightmare. Or something as simple as an app for a a College which allows the teacher to shut down all the interfaces for the switches which are in their classroom, and then to turn them all back on with a click of the button.

No login to the NMS.

No call to the help desk.

Just shutting down the ports when the students aren’t listening, and turning them back on when it’s time to work.

What about you guys? HP has given you some color. What are YOU going to paint?

HP IMC’s New eAPI

Now that I got rum-pooh out of my system…  on to a slightly more technical post.

Not sure if anyone of you caught the recent announcements about the new eAPI from HP’s Intelligent Management center.  In a nutshell, this is a RESTful API which allows programatic access to almost all ( maybe all?) of the IMC functions through an HTTP(s) interface.

Now I’m not a programer at all, but I like to think I have a working knowledge of programing logic. At least enough to give a half-decent programmer enough information to get the job done.

So when I had a co-worker present me with a problem earlier this week, I thought “Hey, I wonder what this new eAPI can really do?”.  I did mention I’m not a programer right?  After this little exercise, I’m thinking I might just have to pick up some scripting skills this year. 🙂

So what’s the value of the eAPI? It functionally allows IMC to act as the progamatic upper layer APIs and abstracts the actual management task from the underlying hardware devices.

In less complicated terms; it means that a program can say  “Hey, IMC change the VLAN on this port” and IMC, assuming IMC actually supports that particular device, it will change the VLAN on that port.

INDEPENDANT OF THE ACTUAL VENDOR

Yup. That’s right.  IMC doesn’t care if that device is a HP 5500EI ( comware ), a HP 3800 ( procurve), or even a Cisco Catalyst 3560.  From the perspective of the developer on the other side. It’s as simple as “Hey, IMC change the VLAN on this port”.

So the actual challenge I was given was the following.

” A customer wants to take a bar-code reader, scan in the MAC address of a device, plug it into the network, push a button and then have IMC automatically put it in the right VLAN”.

Now first I had to break that down into the various components.

1) Bar-code reader scans the MAC address

2) Program has to capture the MAC address for use in the %mac-address% variable in the script.

3) Find the device in the network

Hmmm… this could be more difficult than I thought.

So, I need to mock this up, so I break out to a windows CMD prompt.

Ping a known address ( my Synology NAS — LOVE THIS PRODUCT ).  And then put do a arp -a  to get the following output

10.101.0.51           00-11-32-10-03-8b

Now if I was using the IMC web-interface, I would just use the   Resource-Terminal Access-Real-Time Location feature which will, you guessed it, locate a host in real-time using the mac-address or the IP address.

Hmmm… that kinda sucks for output for the script to leverage.  So I went and looked at the eAPI documentation and came out with this little baby

The eAPI call is the following  ( if you wanted to search for an IP address you would use type=2 instead of 1 )

( Don’t click this, it won’t take you anywhere)

http://10.101.0.201:8080/imcrs/res/access/realtimeLocate?type=1&value=00-11-32-10-03-8b

The return is the following

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

<list>

<realtimeLocation>

<locateIp>00:11:32:10:03:8b</locateIp>

<deviceIp>10.101.0.221</deviceIp>

<ifDesc>GigabitEthernet1/0/21</ifDesc>

</realtimeLocation>

</list>

Yup. Good old XML. Easy to apply transforms or grab variables.  Programmers love this stuff.

4) So now I have the device IP (switch) that this is plugged into, and the ifDesc which is the actual interface it’s located on. So now I have to figure out how to apply the VLAN to this interface. So I break out the trusty eAPI documentation and start looking for the VLAN section.

Hmmm… I have the devIP and the ifDesc.. not the devID and the ifIndex

note to self: Feedback to the developers to have the first command return the devID and the ifIndex variables

So now I have to find the devID and the ifIndex for that devIP and intDesc

5a) Now if I was on the trusty IMC web interface, I would go to the device resource page… hmmm… that doesn’t appear to be there. I guess instead, let’s go to the eAPI documentation and look for something that looks like a dev query.

Yup. It’s actually called devquery. And it looks like I can filter based on the device IP.

Cool. So now I can search for the specific device IP and hope we get the devID variable back that we need for the VLAN call.

http://10.101.0.201:8080/imcrs/plat/res/device?ip=10.101.0.221

The return is the following

  <?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

<list>

<device>

<id>15</id>     ———————————————- This is the device ID that we need to reference later

<label>HP_5500EI</label>

<ip>10.101.0.221</ip>

<mask>255.255.255.0</mask>

<status>1</status>

<statusDesc>Normal</statusDesc>

<sysName>HP_E5500EI</sysName>

<contact>HP Montreal</contact>

<location>Marlborough, MA 01752 USA</location>

<sysOid>1.3.6.1.4.1.43.1.16.4.3.36</sysOid>

<sysDescription>HP_5500EI</sysDescription>

<devCategoryImgSrc>Switch</devCategoryImgSrc>

<topoIconName>stack</topoIconName>

<categoryId>1</categoryId>

<symbolId>1022</symbolId>

<symbolName>HP_5500EI</symbolName>

<symbolType>3</symbolType>

<symbolDesc>HP_5500EI</symbolDesc>

<symbolLevel>3</symbolLevel>

<parentId>1003</parentId>

<typeName>3Com S4800G PWR 24-Port</typeName>

<mac>00:1e:c1:dc:fc:01</mac>

<link op=”GET” rel=”self” href=”http://10.101.0.201:8080/imcrs/plat/res/device/15” />

</device>

</list>

There it is.  DevID is  “15”.

5b) So now we need to figure out that ifIndex value associated with <ifDesc>GigabitEthernet1/0/21</ifDesc> that we pulled above.

If I was in the webinterface, I would simply go to the device ( 10.101.0.221 ), click on the interface list, click on interface Gig 1/0/21 and I would pull out the ifIndex from the interface…

But again, those programmers don’t want HTML, they want an easy XML output that they can play with. So let’s find that…

http://10.101.0.201:8080/imcrs/plat/res/device/15/interface?start=1&size=100

This returns a whole bunch of data for all the interfaces on the switch, but I’m sure that any decent programmer can write a regex expression to only return the one who’s ifDesc value is for Gig 1/0/21, right?

<interface>

<ifIndex>21</ifIndex>      ——————————————–In this case, the ifindex value is the same as the port number. That’s not always going to be true. This is the other variable for the set VLAN

<ifType>6</ifType>

<ifTypeDesc>ETHERNETCSMACD</ifTypeDesc>

<ifDescription>GigabitEthernet1/0/21</ifDescription>

<adminStatus>1</adminStatus>

<adminStatusDesc>Up</adminStatusDesc>

<showStatus>2</showStatus>

<statusDesc>Down</statusDesc>

<operationStatus>2</operationStatus>

<operationStatusDesc>Down</operationStatusDesc>

<ifspeed>10000000</ifspeed>

<appointedSpeed>-1</appointedSpeed>

<ifAlias>GigabitEthernet1/0/15 Interface</ifAlias>

<phyAddress>00:1e:c1:dc:fc:4f</phyAddress>

<mtu>1522</mtu>

<lastChange>4 day(s) 21 hour(s) 39 minute(s) 50 second(s) 990 millisecond(s)</lastChange>

<lastChangeTime>42359099</lastChangeTime>

<filterTrapStatus>0</filterTrapStatus>

</interface>

So now we have the DevID     15     and the ifIndex    for the actual interface where that MAC-address is located.

So let’s go back to the set that VLAN

Let’s assume that you wanted to put the device in VLAN 20,   you would run the following

http://10.101.0.201:8080/imcrs/vlan/20?devId=15&ifIndex21

That’s about it for the task. Now any decent programmer is going to have to put in some checking and error handling, for instance, you might want to check whether or not that VLAN actually EXISTS on the switch. ( Can’t put a VLAN on a port if the VLAN doesn’t exist on the switch, right? ) or maybe return an error if the MAC-address is actually seen on two interfaces, but in a nutshell that’s it.

note: I would also suggest that the dev actually bounce the port to make sure that the device hasnt’ gotten locked in with a DHCP address on the wrong subnet.

/plat/res/device/{deviceId}/interface/{ifIndex}/down   to down the interface

so for use that would be  /plat/res/device/15/interface/21/down

and then immediately do a

/plat/res/device/{deviceId}/interface/{ifIndex}/up

or again for us  /plat/res/device/15/21/up

Now whether the actual switch commands are “switchport acces vlan 20 ”  or ” port access vlan 20 ” or some other variation on a theme doesn’t actually matter to your devOps team. They just write the code to follow the steps and IMC and the eAPI will take care of the rest.

Pretty cool stuff. 🙂

@netmanchris