Implenting Idempotency using HPE IMC

 

Try saying that five times fast.

 

What if those VLANS already exist?

There’s a concept called idempotency which can be loosely explained as

Make sure it’s like this. If it’s not like this, make it like this. If it’s already like this. Don’t do anything

Essentially, it’s a way to declare the desired configuration state of whatever it is you’re trying to configure. If the configuration state of that server, or switch or router is already in that state, than just leave it alone.

It’s a way to ensure that configuration drift doesn’t happen.

So if there’s some rabbid network administrator with a console cable running around laughing maniacly as they randomly changes things… this will help you keep them in check.

jack photo

 

Idempotent VLANs

So we’re going to look at the last example here where we did the following:

  • grabbed the jinja template for vlans directly from a GIThub repository
  • grabbed the desired vlans file directly from a GIThub repository
  • renderd the Jinja template using the values from the vlan file to get our final config
  • used the pyhpeimc library to push the commands through the executecmd RESTful API
 

Import Libraries

You know the drill here, right? Like in all the other examples, and pretty much every useful python script on the planet, we need to first import the specific libraries that we need to help us achieve whatever outcome it is that we want to perform.

In [2]:
import requests
import yaml
import time
from pyhpeimc.auth import *
from pyhpeimc.plat.device import *
from pyhpeimc.plat.icc import *
from pyhpeimc.plat.vlanm import *
auth = IMCAuth("http://", "10.101.0.203", "8080", "admin", "admin")
#auth = IMCAuth("http://", "kontrolissues.thruhere.net", "8086", "admin", "admin")
 

Download VLANs list from Github

Just like in the last blog post, we’re going to download the VLAN’s directly from the GIThub account. This ensures that we’ve got control versioning in place, as well as all the collaborative multi-user goodness that GIThub gives us. If you’re not already using it for SOMETHING. You should be asking yourself “why”?

In [3]:
desired_vlan_list = yaml.load(requests.get('https://raw.githubusercontent.com/netmanchris/Jinja2-Network-Configurations-Scripts/master/vlans.yaml' ).text)
 

As we’re just starting to play around with this, it’s always good to ensure that what we THINK we’ve got is what we’ve actually got. We’re going to now print out the contents of the GITHub file to make sure we know exactly what VLANs are actually in there.

In [4]:
print (yaml.dump(desired_vlan_list['vlans'], indent = 4))
 
- {vlanId: '1', vlanName: default, vlanStatus: '1'}
- {vlanId: '2', vlanName: TenantABC, vlanStatus: '1'}
- {vlanId: '3', vlanName: management, vlanStatus: '1'}
- {vlanId: '10', vlanName: mgmt, vlanStatus: '1'}

 

Gather just the VLAN IDs

If this was my production network, I’d probably be doing more than just checking the VLAN ID, but for our purposes, I’d like to do a quick and dirty “Does a VLAN with this ID exist or not on the device I’m looking at” check.

I’m not currently doing 802.1x identify based networking usng the VLAN name as the deployment key, so this is going to work just fine for me.

I’m going to do a list comprehension to pull out just the VLAN IDs from the YAML file above and store them in the variable called desired vlans_ids. This will setup the list of things VLAN IDs I want to compare the current state to. Make sense?

In a nutshell, this new list will let us compare the desired VLAN IDs to the existing VLAN IDs fairly easily.

In [14]:
desired_vlan_ids = [vlan['vlanId'] for vlan in desired_vlan_list['vlans']]
desired_vlan_ids
Out[14]:
['1', '2', '3', '10']
 

Get Current VLANs on Target Device

Now that we’ve got the desired list, we need to figure out the existing list of VLANs on the target device. This is a two step process

  • get the device ID of the target device using the get_dev_details function and look at the value in the id key.
  • run the get_dev_vlans function usng the devid from step one as the inut value to designate the target device.
In [15]:
devid = get_dev_details('10.20.10.10', auth.creds, auth.url)['id']
dev_vlan_list = get_dev_vlans(devid, auth.creds, auth.url)
 

What do we have here?

As with the other steps, we’ll stop here and take a look to see exactly what’s currently on the device to make sure that our code is working as desired. In a production environment, we would have to trust that this was all working properly, and make sure that we had all the appropriate tests built into our code to make sure that the trust was well deserved.

In [16]:
print (yaml.dump(dev_vlan_list, indent = 4))
 
- {vlanId: '1', vlanName: default, vlanStatus: '1'}
- {vlanId: '5', vlanName: DoesntBelong, vlanStatus: '1'}

 

Add Desired VLANs to Target Device

Now that we’ve got the current and desired state of the VLANs on the device. We need to figure out how to make them match.

For the first step, we will need to figure out how to create and any of the missing VLANs and push them to the target device.

Thankfully, there’s a create_dev_vlan function in the pyhpeimc library that allows us to push VLANs to the device directly using an API without having to use the CLI. No CLI commands is a good thing here, right?

This means that we will not have to worry about vendor specific syntax and can focus on what really matters which is the VLAN IDs, names, and descriptions. Everything else is just details.

In [17]:
help (create_dev_vlan)
 
Help on function create_dev_vlan in module pyhpeimc.plat.vlanm:

create_dev_vlan(devid, vlanid, vlan_name, auth, url)
    function takes devid and vlanid vlan_name of specific device and 802.1q VLAN tag and issues a RESTFUL call to add the
    specified VLAN from the target device. VLAN Name MUST be valid on target device.
    :param devid: int or str value of the target device
    :param vlanid:int or str value of target 802.1q VLAN
    :param vlan_name: str value of the target 802.1q VLAN name. MUST be valid name on target device.
    :return:HTTP Status code of 201 with no values.

 

Creating our Add VLANs function

Now that we understand how the create_dev_vlans function works. We’ll create a new function which will take a full list of VLANs in the desired_vlans_list and check if the it already exists in the dev_vlan_ids variable that we created above. If it already exists; we do nothing. If it doesn’t exist, we will add it.

Just for giggles, I also included a small timer which will allow us to see how long it actually takes for this function to run.

In [18]:
def add_vlans():
    start_time = time.time()
    for vlan in desired_vlan_list['vlans']:
        if vlan['vlanId'] in dev_vlan_ids:
            pass
        else:
            print ('adding vlan ' + str(vlan['vlanId']))
            create_dev_vlan(devid, vlan['vlanId'], vlan['vlanName'], auth=auth.creds, url=auth.url)
            
    print("Operation took --- %s seconds ---" % (time.time() - start_time))
 

Adding the VLANs

Now we simply run the function we defined above to add the VLANs to our target device. You can see from the output below that this took a whopping 0.43 seconds to add the missing three VLANs to the device.

In [19]:
dev_vlan_ids = [ vlan['vlanId'] for vlan in (get_dev_vlans(devid, auth.creds, auth.url))]
add_vlans()
get_dev_vlans(devid, auth.creds, auth.url)
 
adding vlan 2
adding vlan 3
adding vlan 10
Operation took --- 0.43477892875671387 seconds ---
Out[19]:
[{'vlanId': '1', 'vlanName': 'default', 'vlanStatus': '1'},
 {'vlanId': '2', 'vlanName': 'TenantABC', 'vlanStatus': '1'},
 {'vlanId': '3', 'vlanName': 'management', 'vlanStatus': '1'},
 {'vlanId': '5', 'vlanName': 'DoesntBelong', 'vlanStatus': '1'},
 {'vlanId': '10', 'vlanName': 'mgmt', 'vlanStatus': '1'}]
 

Let’s do that again

Now we run the same thing again, but this time all the VLANs already exist so there’s no need to add them. The timer function tells us this took an amazing 3.814e-06 seconds. If memory serves, I think that’s 5 pico seconds.

Let’s run it again a few times to see if that stays the same.

In [20]:
dev_vlan_ids = [ vlan['vlanId'] for vlan in (get_dev_vlans(devid, auth.creds, auth.url))]
add_vlans()
 
Operation took --- 3.814697265625e-06 seconds ---
In [23]:
dev_vlan_ids = [ vlan['vlanId'] for vlan in (get_dev_vlans(devid, auth.creds, auth.url))]
add_vlans()
 
Operation took --- 7.152557373046875e-06 seconds ---
In [24]:
dev_vlan_ids = [ vlan['vlanId'] for vlan in (get_dev_vlans(devid, auth.creds, auth.url))]
add_vlans()
 
Operation took --- 3.814697265625e-06 seconds ---
 

Remove Undesired VLANs from Target Device

Now that we’ve added all the VLANs that SHOULD be there, we need to make sure that we get rid of those “undesirables”. we want the state to be exactly what was defined in the GITHub file, no more, no less, right?

We’ll go back to the pyhpeimc library which has a delete_dev_vlans function pre-built for our usage.

This time we’ll do the exact opposite of above. Instead of adding VLANS which aren’t in the list; we’re going to be removing VLANS which aren’t in the list.

In [25]:
help (delete_dev_vlans)
 
Help on function delete_dev_vlans in module pyhpeimc.plat.vlanm:

delete_dev_vlans(devid, vlanid, auth, url)
    function takes devid and vlanid of specific device and 802.1q VLAN tag and issues a RESTFUL call to remove the
    specified VLAN from the target device.
    :param devid: int or str value of the target device
    :param vlanid:
    :return:HTTP Status code of 204 with no values.

In [26]:
def del_vlans():
    start_time = time.time()
    for vlan in get_dev_vlans(devid, auth.creds, auth.url):
        if vlan['vlanId'] not in desired_vlan_ids:
            print ("Deleting vlan " + vlan['vlanId'])
            delete_dev_vlans(devid, vlan['vlanId'], auth.creds, auth.url)
        else:
            print ('Not touching VLAN ' + str(vlan['vlanId']))
    print("Operation took --- %s seconds ---" % (time.time() - start_time))
In [31]:
del_vlans()
 
Operation took --- 5.9604644775390625e-06 seconds ---
Not touching VLAN 1
Not touching VLAN 2
Not touching VLAN 3
Not touching VLAN 10
Operation took --- 0.1680889129638672 seconds ---
 

And again!

Running this the first time took 0.19 seconds. But, since we’ve not got our target device in the desired state. We should now be able to run the command again and see the time come down considerably as, this time, we’re checking the device and finding out there’s nothing to do.

Let’s take a look:

In [29]:
del_vlans()
 
Not touching VLAN 1
Not touching VLAN 2
Not touching VLAN 3
Not touching VLAN 10
Operation took --- 0.07348895072937012 seconds ---
 

Putting it together

Now that we’ve created both functions, let’s run them both at the same time.

In [32]:
add_vlans()
del_vlans()
 
Operation took --- 5.0067901611328125e-06 seconds ---
Not touching VLAN 1
Not touching VLAN 2
Not touching VLAN 3
Not touching VLAN 10
Operation took --- 0.16545391082763672 seconds ---
 

Embracing the possibilities

So you might be saying “so what?” you just added some vlans to a single switch. With a bit of tweaking, we could easily have the add_vlans() and del_vlans()functions take the IP address of a target device as an input to the function. In this case, we could deploy the VLANS to ALL of the target devices in a specific group, or branch, or campus, or the entire network if we really wanted. That’s the beauty of a little idea.

You can see how the automation of a single small task can quickly save you a lot of time, not to mention the fact that there is no possiblity for human error at the CLI and you will have a predicatable outcome from the centralised YAML file that’s under version control.

Not bad for a network guy, right?

As always, comments or questions are more than welcome. It’s also cool if you just wanted to say “hi”. 

@netmanchris

 

Cleaning up After Ourselves

For those of you following along at home. I have been running this demo a lot lately so I wrote this additional code to get the devices back into the original state. Making it much easier to just run through the whole ipython notebook and perform the same demo in a predicatble manner every time.

I’ve included the code here in case anyone else finds it useful.

In [33]:
create_dev_vlan(devid, '5', 'DoesntBelong', auth.creds, auth.url)
remove_vlans = [ vlan['vlanId'] for vlan in desired_vlan_list['vlans']]
print (remove_vlans)
for i in remove_vlans:
    delete_dev_vlans(devid, i, auth.creds, auth.url)
 
['1', '2', '3', '10']
Unable to delete VLAN.
VLAN does not Exist
Device does not support VLAN function
Vlan deleted
Vlan deleted
Vlan deleted
In [34]:
get_dev_vlans(devid, auth.creds, auth.url)
Out[34]:
[{'vlanId': '1', 'vlanName': 'default', 'vlanStatus': '1'},
 {'vlanId': '5', 'vlanName': 'DoesntBelong', 'vlanStatus': '1'}]
 
Advertisements

More Jinja – Working with YAM as an Input

 

Jinja2 Simple YAML Example


We’re now going to take a look at grabbing a file from the hard drive written in YAML syntax. YAML is arguably the most human readable data serialization format which makes it really easy for coders and non-coders alike to work with.

We’re going to build on the last Jinja2 example. Instead of creating the templates and variables directly in python. We’re going to load them instead from files on our computer.

This may seem like a small detail, but this allows us to deconstruct the building of our configurations, meaning that different people can be responsible for different components of the configuration. As with anything, if you can break a complex process down into several smalller less complex tasks, the whole thing starts to feel easier.

Loading Libraries

We’ll start by loading the required libraries

In [2]:
import yaml
from jinja2 import Environment, FileSystemLoader, Template
 

Set the Environment

Essentially, this set’s the path which will define the directory where the templates will be loaded from. In this case, I’m setting it to load from the same directory.

In [3]:
ENV = Environment(loader=FileSystemLoader('./'))
 

Printing the YAML File

In this step we’re going to use the with open() as file: way of opening the file and just printing it out so you can have a look at what the YAML syntax actually looks like. The advantage of using the with open as… way of opening the file is that there’s no need to go back and close the file when you’re done. May not sound like a big deal, but trust me, files don’t like to be orphaned in an open pyton variable somewhere.

If you’re comfortable with configuring a network device by hand, you can PROBABLY figure out exactly what these variables are doing. The hostname istestswitch, the SNMP configuration are in the snmp section. The read string is supersecret, the vlans are in the vlans section, etc…

Really easy to understand exactly what’s going on here, right?

In [15]:
with open('simple_config.yaml') as file:
    print (file.read())
 
hostname: testswitch

ip: 10.101.0.221


snmp:
    read: supersecret
    write: macdonald
    syscontact: admin.lab.local
    syslocation: lab
    trap:
    - {target: 10.101.0.200}
    - {target: 10.101.0.201}
    - {target: 10.101.0.202}



vlans:
- {description: management vlan, id: '10', name: management}
- {description: users vlan, id: '15', name: users}
- {description: phones vlan, id: '16', name: phones}
- {description: servers vlan, id: '20', name: servers vlan }
 

Loading the YAML File

In this next step, we’ll use the same with open as… way of opening the file, except rather than just reading the file this time, we’re going to use the yaml.loadmethod to load the contents of that file into a python variable called simple.

The nice thing is that the YAML library takes care of all the hard work for us. Automation is supposed to make things easier, right?

In [12]:
with open("simple_config.yaml") as simple:
    simple =  yaml.load(simple)
 

Looking at the simple variable

So now that we’ve loaded the contents of simple.yaml into the simple variable, let’s take a look at what python sees. We’ll run the type() command first to see what kind of object simple has been created as, and then we’ll print out the contents of simple so you can see how the YAML library has transformed it from the text in the YAML file to something which is easier to work with in python.

In [16]:
type(simple)
Out[16]:
dict
In [11]:
simple
Out[11]:
{'hostname': 'testswitch',
 'ip': '10.101.0.221',
 'snmp': {'read': 'supersecret',
  'syscontact': 'admin.lab.local',
  'syslocation': 'lab',
  'trap': [{'target': '10.101.0.200'},
   {'target': '10.101.0.201'},
   {'target': '10.101.0.202'}],
  'write': 'macdonald'},
 'vlans': [{'description': 'management vlan',
   'id': '10',
   'name': 'management'},
  {'description': 'users vlan', 'id': '15', 'name': 'users'},
  {'description': 'phones vlan', 'id': '16', 'name': 'phones'},
  {'description': 'servers vlan', 'id': '20', 'name': 'servers vlan'}]}
 

Templates


Printing the simple_cisco template

In this step we’re going to take a look at the simple_cisco.j2 template that I’ve created. This is a very simple configuration just to show the power of jinja for making your network configurations.

If you look closely, you’ll see a couple of {% for…%} tags in here. This is a control structure called an interator, commonly known as a For loop. Essentially it’s saying, take the list of things and do this one action to each of the things in the list.

One thing to note here is that j2 is not a jinja2 specific file extention, but just something that I, and many others I’m sure, use to designate their template files.

In [28]:
with open('simple_cisco.j2') as file:
    print (file.read())
 
#hostname config
hostname {{ simple['hostname'] }}
#vlan config
{% for vlan in simple['vlans'] -%}
vlan {{ vlan['id'] }}
    name {{ vlan['name'] }}
    description {{ vlan['description'] }}
{% endfor %}#snmp config
snmp-server community {{ simple['snmp']['read'] }} RO
snmp-server community {{ simple['snmp']['write'] }} RW
snmp-server ifindex persist
snmp-server location {{ simple['snmp']['syslocation'] }}
snmp-server contact {{ simple['snmp']['syscontact'] }}
{% for trap in simple['snmp']['trap'] -%}
snmp-server host {{ trap['target'] }}  public
{% endfor %}
 

Rendering the simple_cisco template

In this last step of working with the simple_cisco template, we’re going to now pass this through the jinja rendering engine. Because we’ve got dynamic parts in the template, we’re going to have to supply a source for the variables to fill the dynamic part in. If you look below, we’re saying that anytime you see the wordsimple in the template, you should look in the variable simple we created above and see if there’s the appropriate information there to fill it in.

In simpler terms, we’re going to take the simple.yaml file we loaded above as the input values into this template.

We then render the template and…

In [29]:
template = ENV.get_template("simple_cisco.j2")
print (template.render(simple=simple))
 
#hostname config
hostname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp config
snmp-server community supersecret RO
snmp-server community macdonald RW
snmp-server ifindex persist
snmp-server location lab
snmp-server contact admin.lab.local
snmp-server host 10.101.0.200  public
snmp-server host 10.101.0.201  public
snmp-server host 10.101.0.202  public

 

Look familiar right? Minimal typing. Alll the VLANS are there, etc… and the best thing is if I run this a thousand times, it will always come out the same way.

 

But I have a multi-vendor network!!!!

This is the true power of jinja for me. I happen to run multiple vendors in my lab, but I’d like to have the ability to drive all of the configurations from a central location to make sure that they all have the same vlans, usernames and passwords, snmp strings, etc…

So now let’s take a look at running another vendor’s template using the same simple.yaml file as the input source.

 

Printing the simple_comware template

In this step we’re going to take a look at the simple_comware.j2 template that I’ve created. This is a very simple configuration just to show the power of jinja for making your network configurations.

You’ll notice that it’s very close to the simple_cisco.j2 file shown above. The real difference is the parts outside of the jinja2 variables. HPE Networking’s comware devices use the keyword sysname instead of hostname. They use the keyword snmp-agent instead of snmp-server. Minor differences in the syntax, but the actual values are exactly the same for both devices.

In [30]:
with open('simple_comware.j2') as file:
    print (file.read())
 
#sysname config
sysname {{ simple['hostname'] }}
#vlan config
{% for vlan in simple['vlans'] -%}
vlan {{ vlan['id'] }}
    name {{ vlan['name'] }}
    description {{ vlan['description'] }}
{% endfor %}#snmp_config
snmp-agent
snmp-agent community read {{ simple['snmp']['read'] }}
snmp-agent community write {{ simple['snmp']['write'] }}
snmp-agent sys-info contact {{ simple['snmp']['syscontact']  }}
snmp-agent sys-info location {{ simple['snmp']['syslocation'] }}
snmp-agent sys-info version all
 

Rendering the simple_comware template

In this step, we’re going to render and print the template using the same simple variable that we used for the simple_cisco.j2 template above. Because of the difference in the simple_comware.j2 templates, it will render with the proper syntax for the HPE devices.

In [31]:
template = ENV.get_template("simple_comware.j2")
print (template.render(simple=simple))
 
#sysname config
sysname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp_config
snmp-agent
snmp-agent community read supersecret
snmp-agent community write macdonald
snmp-agent sys-info contact admin.lab.local
snmp-agent sys-info location lab
snmp-agent sys-info version all
 

Wrap Up

I’m sure you’ll agree this is a much better way of creating your configurations than grabbing a console cable and typing them all out by hand, right? But there’s still a lot of room for improvement! Currently, we’re going to render the template and then cut and paste them into a console cable. So although we’ve made some gains… it could be better.

We’ll take a look at that next time.

@netmanchris

 

Intro to Jinja2

 
 

What is Jinja2

Jinja2 is a templating language that was originally used as part of the Flask python web framework. From the Jinja2 website

Jinja2 is a full featured template engine for Python. It has full unicode support, an optional integrated sandboxed execution environment, widely used and BSD licensed

It was originally developed to help automatically generate HTML dynamically as part of the flask framework, more on that in another post, but it can also easily be used to help us generate our configuration files for our infrastructure devices.

This is going to be a very simple introduction to a few of the basic concepts of that jinja uses which, hopefully, will help to understand how Jinja can be used as a first step down the road of gaining automation skills.

We’ll take a look at a developing some intuition on how Jinja2 can be used to create basic network infrastructure device configurations. This is definitly not the modern method of interfacing directly into the control/data/management plane of devices using APIs, but it’s definitely a step in the right direction on understanding how a bit of code can help make your life better.

Prereqs

I’m assuming you’ve already got python installed on your system. You’re also going to need to run the pip install jinja2 command from a terminal window to get the latest version of jinja2 which should work just fine here.

 

Learning by Example

In this section we’ll start with a small example on how to create a few VLANs using the typical syntax from a modern networking OS. In this case, I used the HPE Comware syntax, but it would be easy enough to create this using a Cisco or Juniper configuration and you’re encouraged to try to get this working with your own network vendor.

 

Import required libraries

First We’ll import the required modules from the Jinja2 library. This is pretty much stolen directly from the jinja2 docs.

In [1]:
from jinja2 import Environment, FileSystemLoader, Template
 

Creating the VLANS

For this example, we’re going to create a python list of dicts which contains six different VLANS as listed in the tabel below.

A python dictionary is just a key-value pair, where the value for a specific key in the dictionary can be accessed using the key name.

  • Name: Name of the VLAN
  • Description: Descrition of the VLAN
  • VLAN ID: Dot1q VLAN ID.
Name Description VLAN ID
Management Management VLAN 10
Users Users VLAN 15
Phones Phones VLAN 16
Servers Servers VLAN 20
Mobility Mobility VLAN 30
Guest Guest VLAN 40
In [5]:
vlans_list = [{'name': 'management', 'description': 'management vlan', 'id': '10'},
         {'name': 'users', 'description': 'users vlan', 'id': '15'},
         {'name': 'phones', 'description': 'phones vlan', 'id': '16'},
         {'name': 'servers', 'description': 'servers vlan', 'id': '20'},
         {'name': 'mobility', 'description': 'mobility vlan', 'id': '30'},
         {'name': 'guest', 'description': 'guest vlan', 'id': '40'},
         {'name': 'rob', 'description': 'guest vlan', 'id': '45'}
         ]
 

VLAN Jinja2 Template

In this step, we’re going to create a variable called text_file which will contain the content of a jinja2 template. This is a basic python string object which means, at this point, it’s just a bunch of text.

In normal circunstances, we would actually be reading this template from a file located on the hard drive, but for our purposes today, we’ll just put the templatein by hand.

What makes Jinja2 powerful is the control structures that allow it to perform programatic operations. In this example, we’re creating a For loop.

Following the code we will each vlan in the vlans object we created above and then render the template using the ‘id’ key for the first variable, the ‘name’ key for the second variable, and the ‘description’ key for the last variable.

Hopfully, this makes sense, but if not, just hold on and it should become clear before the end.

In [6]:
text_file = ('''
#vlan config
{% for vlan in vlans -%}
vlan {{ vlan['id'] }}
 name {{ vlan['name'] }}
 description {{ vlan['description'] }}
{% endfor %}''')
 

If I was to write the same as a traditional python iterator it would look something like this. You can see how they are related I hope?

In [7]:
for vlan in vlans_list:
    print ('''vlan ''' +vlan['id']+
           '''\n name '''+vlan['name']+
           '''\n description '''+vlan['description'])
 
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers
    description servers vlan
vlan 30
    name mobility
    description mobility vlan
vlan 40
    name guest
    description guest vlan
vlan 45
    name rob
    description guest vlan
 

That’s a lot of work typing isn’t it?

You could ask

That’s more typing than I would do by hand? Why would I use this?

Great question. The point of automating anything is to cut down on the repetitive effort it takes to accomplish a given goal. In this case, we can simply count the number of key strokes it would take to create a single new VLAN on a switch.

In [6]:
count_chars = "vlan', 'name', 'description"
keystrokes = len(count_chars)
print (keystrokes)
 
27
 

Now let’s pretend we had to type that 10 times

In [7]:
keystrokes * 10
Out[7]:
270
 

Or maybe we had to create 100 VLANs.

In [10]:
keystrokes * 100
Out[10]:
2700

Ir maybe we had to create the full 4094 VLANs available

In [11]:
keystrokes * 4094
Out[11]:
110538
 

Not sure about you, but if I don’t have to type 110,000 keystrokes, my fingers will love me at the end of the day. Not to mention the fact that it’s also repeated perfectly every single time, not a single typo in there.

 

Create the Template Object

Now that we’ve created the text_file string object, we need to transform it into a jinj2 template which will allow us to then render it. We will create a new object called vlan_template and assign an instance of the Template class using the text_file contents as the input.

In [12]:
vlan_template = Template(text_file)
 

Make the Magic Happen

We will now use the render method on the vlan_template that we created above. We have a single argument to pass into the function. In this case we are passing the vlans_list list of dictionaries we create above in to the function as the vlans variable.

In [13]:
vlan_template.render(vlans=vlans_list)
Out[13]:
'\n#vlan config\nvlan 10\n    name management\n    description management vlan\nvlan 15\n    name users\n    description users vlan\nvlan 16\n    name phones\n    description phones vlan\nvlan 20\n    name servers\n    description servers vlan\nvlan 30\n    name mobility\n    description mobility vlan\nvlan 40\n    name guest\n    description guest vlan\n'
 

Hmmm What happened there?

That doesn’t look like a configuration file does it? The output of this file is actually a python string object. In python, we need someway to represent a carriage return (enter-key) and the \n just happens to have that honour.

Instead of running the template rendering directly, we can instead capture the output into a string object which we will then pass to the print command.

In [14]:
rendered_template = vlan_template.render(vlans=vlans_list)
In [15]:
print (rendered_template)
 
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers
    description servers vlan
vlan 30
    name mobility
    description mobility vlan
vlan 40
    name guest
    description guest vlan

 

Clear?

Hopefully, this has shown you a bit of how a basic jinja control structure, like a For loop, can be used to cut down on a lot of key strokes, increase the accuracy of the configurations and help to streamline the operations.

In the next post, I’ll look at loading YAML files directly into python and using their contents as input into some more advanced jinja2 templates.

 

I use SNMP SETs and I’m not afraid to admit it.

Do you remember back in CCNA school when we learned all sorts of great things that we very rarely followed. One of the favourites was that we are supposed to put meaningful descriptions on all of our interfaces so we know what the other side is connected to.

How many people actually follow that advice?

Yeah, I never do it either. There’s always just too many things on the list that need to get done and it seems like that extra 5 seconds it would take me to update the description to the interface just doesn’t seem like it’s worth the effort. Of course, then I later check the port and end up knocking out my XYZ services and cause myself an outage.

This is where a little python and a decent NMS can help to solve a problem.

Understanding ifIndex

Before we get into the code. We need to understand a little about ifIndex values and how they relate to the physical interfaces of the devices. If you’re REALLY interested, you can do some reading in RFC 2863.  But in a nutshell, each interface on a device, whether physical or logical has a specific numeric value assigned to it which is the last digit in the interface statistics that can be seen through the SNMP interfaces. This is commonly known as interfaces group stats.

There are a bunch of different tables in the interface group stats that are used to store specific kinds of information or statistics for the interfaces such as

  • ifNumber (.1.3.6.1.2.1.2.1.0) – Which is the total number of Interfaces
  • ifIndex ( .1.3.6.1.2.1.2.2.1.1. * ) – Which acts as a primary key for all the other interface group stats.
  • ifDescr (.1.3.6.1.2.1.2.2.1.2.* ) – Which is the name of the interface
  • ifType (.1.3.6.1.2.1.2.2.1.3.* ) – Which describes the type of the interface
  • ifMTU ( .1.3.6.1.2.1.2.2.1.4.*) – Which shows the current MTU size configured on the interface
  • ifSpeed (.1.3.6.1.2.1.2.2.1.5.*) – Which shows the current speed of the interface
  • ifPhysicalAddress (.1.3.6.1.2.1.2.2.1.6.*) – Which shows the mac address of the interface
  • ifAdminStatus (.1.3.6.1.2.1.2.2.1.7.*) – Which shows the current admin status of the port
  • ifOperStatus (.1.3.6.1.2.1.2.2.1.8.*) – Which shows the current operational status

There’s a bunch more which you can see here if you’re interested, but one that I found particularly fun was the ifAlias ( .1.3.6.1.2.1.31.1.1.1.18.* ) which actually corresponds to the description command in your friendly neighbourhood network operating system

So when your interface is configured like this

Screen Shot 2015 12 10 at 10 33 12 PM

The ifAlias value for the corresponding ifIndex looks like this

Screen Shot 2015 12 10 at 10 34 30 PM

The Fun Begins

The interesting part about the ifAlias value is that it’s actually a SET-able value through the SNMP interface. That means that if you have a simple piece of python code like the following

It will allow you to run a little command like this >>> set_interface_description(‘10.101.0.221’, ‘1’, ‘Changed This’ ) which will result in the following.

Screen Shot 2015 12 10 at 10 52 41 PM

So this is cool, right? We’ve just programatically changed a single interface description on a interface. We could stop here and you would be left with a

“So what? Why would I go to the trouble of doing that. It’s harder than just typing in the description manually!”

But wait! There’s more…

The really cool part about automating something so simple is that now we have a building block that we can do something with. For those of you who might have seen the HPE Intelligent Management Centre, you may already be aware that the topology map does this really cool thing where it actually creates a table of all of the links within a given custom view and automatically creates link-names for them.

Screen Shot 2015 12 10 at 10 58 57 PM

You’ll notice that the auto-generated Link-Name actually tells me who’s connected to both sides of that link. You’ll also notice that I have the left and right nodes ( with the IP address ), as well as the left and right interfaces. And  “yes”, there is an API for this where it’s all represented in a nice little JSON string which can be easily parsed in your favourite IDE.

Time vs ROI

The reason we don’t label interfaces is that most people feel it’s just not worth the effort to keep them up to date. Things change too often and it’s just too easy to say “it’s not that important” and move on before changing that description.  I’m with you. I’m as guilty as everyone else on this. But with the help of a little code, a good NMS, the entire process is now automated.

I can proudly say that, for now at least, my lab is 100% accurate as far as interface descriptions go. And because the whole thing is totally automated, I simply re-run the script overtime I make some changes.

So for those of you who weren’t aware. Yes, there is someone who actually uses SNMP SETs in the world. 🙂

Presentation Tip: How to Zoom

This is one of those posts inspired by people asking me “Hey! How did you do that Zoom thing while you were presenting?”  I’m not sure how I feel about them not asking me about the actual content, but hey, sometimes that’s how things go.  Hopefully, sharing these tricks will help people pick up a small skill which can make your presentations more accessible for your audience. 

 

Zoom Zoom Zoom

We’ve all been in a situation where you are stuck with an older projector, or just plain don’t have enough screen real-estate to get everything on the screen you need to. The classic eye-chart.  ( By the way,

Note: if you are going to call it an eye-chart, why did you include it in the presentation in the first place?  Call it out as reference material for future review, don’t make people wonder why you put something in the presentation that you intentionally made it difficult for them to see!  #rantover

I’ve started incorporating more and more live demo’s into my presentations, and many of those are showing code examples. Not the easiest thing to see in on a projector. Thankfully, there’s a nice accessibility feature within OSX and Windows that can help your audience with assimilating the information you’re trying to explain, which after all is the whole point of delivering a presentation, right? What’s the message you want to give them and what do you want to leave them understanding at the end. 

 

Note: I’m writing this in an airport on the way to ONUG, so the windows section will get posted in a future update as I’ve only got my MacBook open at the moment. 🙂 

 

OSX

To configure the zooming functions on OSX, it’s pretty easy. Just head over to the settings > accessibility > zoom and turn it on.  I particularly like option Zoom follows the keyboard focus which allows the zoom short-cut keys to focus in on exactly where your cursor is currently located. 

Screen Shot 2015 11 03 at 1 06 27 PM

 

Once you’ve turned on this setting To zoom in, press Command (⌘)-Option (⌥)-Equal Sign (=). To zoom out, press Command (⌘)-Option (⌥)-Minus Sign (-).

Here’s a quick video example of this in action. As you can see, you can zoom in and focus on any area of the screen that you want to draw attention to. 

 

 

 

Windows

This is a placeholder that I will update in the extremely near future.  I do present from both OS’s so it’s nice that both have a very similar feature with almost identical hotkeys.

 

 

Love your audience

Remember at the end of the day, although you may feel like you’re the man because you’re at the front of the room, the most important person in any presentation is each and every individual member of the audience. Anything that we can to to help them understand and consume the message/information that we’re trying to deliver is always going to be welcome. 

 

Automating your NMS build – Part 4 Adding Custom Views

This is part four in a series of using python and the RESTful API to automate the configuration of the HP IMC network management station.  Like with all things, it’s easier to learn something when you’re able to find a good reason to use those skills. I decided to extend my python skills by figuring out how to use python to configure my NMS using the RESTful API. 

If you’re interested, check out the other posts in this series

Creating Operators

Adding Devices

Changing Device Categories

Adding Custom Views

In this post, we’re going to use the RESTful API to programatically add a custom view, and then add devices to that custom view.  For those of you who don’t know, a custom view is simply a logical grouping of devices.  In IMC custom views also form the basis of the topology maps. In fact, a custom view and a topology map are essentially the same object internally,  The difference is just whether you chose to look at it as a list of devices in the normal interface or chose to look at it in the typical topology/visio format which we all know and might-not-love. 

Why might we want to add a custom view programatically you ask? While, the answer to that might be simply that we’re lazy and it’s easier and faster.  Or it might be that you don’t want to have to look through all the different devices, or, as in my case, that you simply want an excuse to extend your python skills.  Whatever you’re reasons are, custom views are something that just don’t get used enough in my opinion. 

Custom views are a great way to be able to zoom in on the status of a specific branch, a geographic area, maybe a logical grouping of devices that support a specific application?  It really doesn’t matter and the best part is that a single device can exist in multiple custom views at the same time, so there’s really no limit to how you put these views together. It all depends on what makes sense to you. 

 

The Code

This code is a little bit more complicated than some of the other examples we’ve looked at. We’re actually going to be using multiple functions together, but the logic should be pretty easy to follow.  I’m sure there are better ways to do this, but this seems to work for me.  I’m sure I’ll be back here in a year going “Why did I write it that way!?!?!?!?!?” but for now, I hope it’s simple enough for someone else to follow and possibly get inspired. 

The main function is really just calling the other functions which we will break down below

Step 1 – Create the Custom View

In this code, we’re going to simply use this small function that I created to gather the name of the custom view ( the view_name variable ) and then use that to create the JSON payload ( the payload variable ).  The other part of the JSON payload is the autoAddDevType variable which is hard-coded to 0.  This could be used to have the system automatically add new devices of a given type, but I”m in interested in doing the automation myself here. The last part of this code will be used as the input for another function, which is the return of the view name.  You can see in the main  function in the the view_name = create_new_view() line. 

Step 2 – Get Custom Views

For the next part, we are going to need to figure out what Id was assigned to this new view, to do this we’re going to have to go through a couple of steps. The first one is to ask the NMS to send us a list of all the known custom views. The following function will request the list, which will be returned as a JSON array, and then convert it over into a python list of dictionaries so we can work with it natively as a python object. You can see this in the main function in the  view_list = get_custom_views() line

Now that we have the view_list which is the list of all the views, we’re going to have to find the ID for the new view that we just created.  We do that by by using the get_view_id() function using the view_name as the input.  Essentially, this will look through each of the views in the view_list that was returned above and let us know when the ‘name’ value is equal to the view_name value that we captured above. Once it’s equal, we then return the ‘symbolId’ which is the internal unique numeric value assigned to this particular custom view. This is the number we’re going to use to identify the view that  that we want to add devices to.  Make sense? Now that we’ve got this number, we’re going to assign it to the object view_id for use later on. 

Note: I actually could have added the devices directly to the view in the original add_custom_view code code above, but then I’d have to write the modify function later if I ever wanted to change or add new devices to the view. I’m trying to follow the DRY ( Don’t Repeat Yourself ) advice here so I just write the modify here and I can then leverage it later without having to re-write the code.  

Step 3 – Generate the Device List

We’re not going to spend too much time on this part as I’m essentially re-using the code from the Changing Device Categories blog.  It’s pretty straight forward. I’m using some user-put to gather a list of devices that we want to add to this specific view and then capture it in the dev_list object.  What’s cool about doing it this way is that the returned list will search through all the IP addresses assigned to your devices, not just the managed address which might come up in the NMS interface where you would normally perform this step. There are ways around that as well, but that’s another blog. 

Step 4 – Add Devices to Existing Custom View

This last step is where things come together. We’re going to run the add_device_to_view(dev_list, view_id) function which is using the dev_list generated in step 3 and the view_id that we captured at the end of step 2 as the inputs.  Essentially, we’re just saying here  “ add all the devices I want to the view I just created “. 

Wrapping it Up

So this is just an example of how you can tie a few pieces of code together to help automate something that might otherwise take you a lot of manual labour. In this case, just a bunch of mouse clicks and depending on the fact that you were able to manually identify all the devices you wanted to add to a specific view.  Personally, I’d rather leave the hard work to the computers and move on to something that requires my brain.

Automating your NMS build – Part 3 Changing Device Categories

In the first couple of posts in this series, we created some operators, then we added some devices.   In this post we’re going to look at something a little more complicated. In this post, we’re going to link a couple of different python functions together to meet the requirement which is to change a device from one category to another.

A bit about Device Categories

For those of you haven’t used HP’s Intelligent Management Centre before, the system automatically categorizes any discovered device, usually based upon SNMP sysobjectid. What this means is that when you run an auto discovery, the majority of your infrastructure will be properly classified right out of the box.

Screen Shot 2015 07 07 at 11 11 51 PM

This works great for devices which are SNMP enabled, as well as some other devices like ESX machines ( Virtual Devices ) which use SOAP as the management protocol, but it fails pretty miserably when dealing with devices which don’t support anything more than PING.  IMC’s default behaviour is to put anything which doesn’t respond to SNMP in the Desktop Category.

IP Phones aren’t smart

I’ve had customers who decided to discover all of their expensive IP phones and suddenly found out that none of them were classified properly. The problem with IP phones is that they are usually pretty stupid devices. Low memory, weak CPUs. Most of the processing/thinking is done by the PBX. I’ve never seen an IP phone that supports SNMP.

In this case we have two choices

  1. Manually change each IP phone from the Desktop category into the Voice category one device at a time.
  2. Use the RESTful API and a bit of code to do it wihile we go have a coffee

Filtering First

So the first thing we need to do is to identify the IP phones from the rest of the devices in the Desktop category. Thankfully, this is where having a well designed network can come in REALLY handy. Most Voice networks are designed so that the IP phones are automatically put into a voice vlan. This means that all phones SHOULD be in the same layer 3 network range.

The first piece of code we need to write is a simple piece of code which will allow the user to identify which of the categories they want to filter by. Although this might sound strange, we actually need to make sure that we only grab the DESKTOP devices in a specific subnet range. Imagine if you accidentally move the router or switch for this subnet into the voice category too. Really sucks leaving when you lose your router, right?

In this piece of code, we’re simply creating a dictionary which creates the link between the categoryId, which is the number IMC internally uses to identify the defined categories and the labels we humans use to identify them.  In a nutshell, this simply prints out the available categories.  Why you ask? Because the human running this script needs to known which categories they want to filter by.

This next performs two different functions

  1. Filters all known devices by the categories listed above
  2. Filters all known devices by an IP range.

Combining the two of them we’re able to easily fine all of the devices that have been classified in the Desktop Category in the L3 subnet of the IP phones and return that as a dictionary which contains, among other things, the device IDs which we’ll need in the next step.

Putting it all together

So the last piece of code is where the magic actually happens.

First we assign the output of the filtering function above into the variable called dev_list.  Essentially, this is a list of devices which meet the search criteria of the filtering function.

In our case, this means the list will consists of all the IP phones in the specific subnet that we filtered.

From there, we use the items in dev_list as input into a for loop which changes them into the new category.  ( see how we use the same print_category function from above? )

Wrapping it up

Hopefully this is a fairly useful example of how putting a few basic python functions together can help to substantially cut down on the amount of time it takes to perform what’s really a simple task. That’s the whole point of automation right?

As I continue learning, I can already see there a bunch of ways to improve this code, but hopefully having a simple working example will help people who are a couple of steps behind me on this journey take another step forward.

If you’re ahead of me on this journey and have suggestions, please feel free to comment!

@netmanchris