Deploying Code to Devices Through your NMS

 
 

note: It’s come to my attention that WordPress is truncating some of my posts so that the right hand side is blocked by the side bar. My apologies for the this. I’ll get it fixed ( or more likely move to GH pages ) as quickly as possible. Thanks for your patience

@netmanchris

 

If you’re luck enough to have an NMS as powerful as HPE IMC then you already have a very capable system which has a ton of APIs that you probably didn’t even know about. IMC isn’t the only NMS which has APIs these days, but it’s the one we’re going to be looking at here.

We’ve spent the last few posts ( herehere, and here running through creating some network configurations through the Jinja2 templating language.

There are at least a couple of immediate benefits to this approach:

  • Consistency in the configuration between devices
  • Accuracy in the commands going into your devices

But the one large draw back is that you’ve still got to cut and paste that configuration into your device somehow, which is not the ideal scenario. We’re trying to get away from touching our devices.

In this post, we’re going to look at taking the rendered configuration and pushing it directly to the desired device through HPE IMC’s RESTful API, refered to as the eAPI in documentation.

Although there used to be a charge for this, HPE recently made some changes and the RESTful API is now included in both the Standard and Enterprise editions of the NMS.

In [2]:
import requests
import yaml
import githubuser
from pyhpeimc.auth import *
from pyhpeimc.plat.device import *
from pyhpeimc.plat.icc import *
from pyhpeimc.plat.vlanm import *
from jinja2 import Environment, FileSystemLoader, Template
 

Loading the templates and values from Git

We’ve gone through this before, so I’m not going to spend much time here going over this. In a nutshell, we’re loading the comware_template and the variables we’d like to use to fill in the template. Again, make sure you’re using the raw URL from Github and not the normal URL or you will end up with the whole HTML structure and not just the content you’re looking for.

In [3]:
comware_template = requests.get('https://raw.githubusercontent.com/netmanchris/Jinja2-Network-Configurations-Scripts/master/comware_vlan.j2').text
gitauth = githubuser.gitcreds() #you didn't think I was going to give you my password did you?
simple = yaml.load(requests.get('https://raw.githubusercontent.com/netmanchris/Jinja2-Network-Configurations-Scripts/master/vlans.yaml', auth=gitauth).text)
cw_template = Template(comware_template)
 

Rendering the template

Here we’re going to take a quick look at the rendered combination of the comware_tempalte and the variables to make sure this is what we want to send during the final push to the device. Automation is great, but it’s going to be a long time before it can replace a human being with knowledge of the environment. Trust… but verify.

In [4]:
my_template = cw_template.render(simple=simple)
print (my_template)
 
#vlan config
vlan 1
    name default
    description default
vlan 2
    name TenantABC
    description TenantABC
vlan 3
    name management
    description management
vlan 10
    name mgmt
    description mgmt

 

Options, Options, Options…

We now have a decision to make. There are a couple of different APIs available to us to push VLANs to the device.

For this example, we’re going to use the executecmd API that allows us to send a series of commands to the device through the HPE IMC REST API.

vlan api

As you can see from the REST documentation, you need to send a JSON object which is a list of the commands that you would type in from the command prompt of the switch.

So there are a couple of things we need to prepare the rendered jinja template into a format that can be sent to the API.

  1. We need to add the command “system-view” to the beginning of the command list.

    system-view on HPE Comware devices is equivalent to the enable + conf t commands using the IOS syntax you’re probably used to

  2. We need to split the giant string that rendering the jinja template gave us into a python list with one command per list item. Thankfully, we can use the python split method to help us through this. We can use the carriage return symbol to identify the end of each line. python identifies the carriage return by the \n characters which is what we’re going to use as the input to the split method.

  3. Once we’ve got those two things done, we simply add the two together and voila!

In [5]:
cmd_list = ['system-view']
cmd_list = cmd_list + my_template.split('\n')
 

Trust but verify

Are you seeing a trend here? If we’re ever going to learn to trust automation, we need to get comfortable that our expectations are met at each step of the journey, so we’re going to take a look at the new cmd_list variable and make sure that

  • it’s a list
  • the first elemend of the list is system-view
  • the rest of the list is one command per element
  • all the commands are in the right order
In [6]:
cmd_list[0:10]
Out[6]:
['system-view',
 '#vlan config',
 'vlan 1',
 '    name default',
 '    description default',
 'vlan 2',
 '    name TenantABC',
 '    description TenantABC',
 'vlan 3',
 '    name management']
 

Sending the commands

So far, other than splitting on the \n, this isn’t much different than what we’ve covered in the other blog posts. Now is where we connect the list of commands we’ve created to the device they are destined for.

The first thing we’re going to do is to create an authentication object that we can use to feed into the requests commands upon sending to the REST API.

In [7]:
auth = IMCAuth("http://", "10.101.0.203", "8080", "admin", "admin")
 
 

Getting the Device ID

The input for the run_dev_cmd is the device ID. This is an internal number that IMC uses to idenitfy that specific device. Thankfully, we’ve also got an RESTful function to get that based on the IP address of the device. To make things a little bit easier on us, we will grab the results of the get_dev_details API and assign the device ID directly to a variable called devid. Once we’ve got the device ID back, this gives us what we need to move on to the next steps.

In [8]:
devid = get_dev_details('10.20.10.10', auth.creds, auth.url)['id']
devid
Out[8]:
'221'
 

Sending the Commands to the target Device

We will now use the run_dev_cmd function from the pyhpeimc library to send the commands directly to the device. You can see that we are using the devidvariable assigned above as the input for the target device. We’re also using the cmd_list variable that containts the list of all the commands that we wish to send to the device.

We’re going to look for the contents of the success response. Which, if we’re lucky, should be true.

In [9]:
run_dev_cmd(devid, cmd_list, auth.creds, auth.url)['success']
Out[9]:
'true'
 

Double Checking the VLANs

Now that we’ve sent the VLANs to the device, the last thing we should be doing is to double check that nothing went wrong in the sending. We’ll use the exact same run_dev_cmd function, but this time, we’ll be sending the display vlan command and looking at the content of the return instead of the success.

In [10]:
cmd_list = ['system-view', 'display vlan']
print (run_dev_cmd(devid, cmd_list, auth.creds, auth.url)['content'])
 
 1(default), 2-3, 5, 10
 

Getting better, right?

So we’ve come a long way in a short time. We’ve

And in this post, we learned how to leverage the first three to deploy configurations directly from code to our devices.

The good part

For those who have done some scripting to device before, you’ll have noticed that using an API provided by an NMS such as HPEIMC makes life much easier. We didn’t have to worry about username and passwords for the individual devces, nor having to worry about deciding what protocol we need to use to connect to the device. The great part about using the NMS as a proxy is that all the credential and protocl negotiations are all handled by the NMS itself, allowing us to get on to the trouble of worrying about what we want to send to our devices and not concerning with how they actually get there.

This is a big step forward, but there are still a couple of small problems that we need to address

Configuration Drift

If you look closely, we’ve actually got an extra VLAN in there. VLAN 5 has been configured on the device, but it’s not in our list of desired_vlans where we have declared exactly which VLANs should be on the target device. This is what is sometimes known as configuration drift. Some people may say

Hey, It’s just an extra VLAN right? That won’t hurt us!

Sorry to respectfully disagree, but this attitude is exactly what causes us issues. This is the death of a thousand cuts. It’s JUST one VLAN, JUST one switch running a differnet version of code, JUST one router that has some unused sub-interfaces on it.

IT’S JUST ONE MORE THING THAT WILL BITE YOU WHEN YOU’RE TROUBLESHOOTING AN ISSUE.

These JUST things are what we sometimes call technical debt. If you can figure out out.

Vendor Syntax

The other problem with this example is that we are bound to a specific vendor’s syntax. If you attmept to run the system-view command on a Juniper/Cisco/Brocade/Extreme/ARISTA device, it’s going to error out. Right? This coule easily be addressed by some conditional logic which figures out which kind of a box it is first and then leverages a specific Jinja template for that vendor, but you can see how this becomes a slippery slope rather quickly.

In the next post, we’re going to look at a way to address both of these issues.

Stay Tuned!

@netmanchris

P.S. As always, comments and questions are more than welcome.

In [ ]:
 
Advertisements

GIT and Jinja – Like Peanut butter and Pickles!

Thanks to @mierdin for point this out. It looks like the wordpress format is causing some strange word-wrap issues. For a better view please click here to see the full post without presentation issues. 

 

Using GITHub to build our Network Configs

As I wrote in this post, one of my goals for this year is to be able to compltely automate the build of my lab environment programatically.

In the last couple of jinja posts, I wrote about the basics of Jinja2 templates and how they can be applied to building network configurations.

In this post, I’m going to take the next step and move those files from my local hard drive out to…

 

duh duh dahhhhhhhhhh

The cloud.

The cloud

 

Before we get started…

We’re going to go over some basics on the tools we’re using to make sure everyone’s on the same page. cool?

What’s GIT?

Git is a widely-used source code management system for software development. It is a distributed revision control system with an emphasis on speed, data integrity, and support for distributed, non-linear workflows. wikipedia

Huh?

GIT is a piece of software that allows you to track changes to files over time.

So what’s GITHub?

“Where software is built Powerful collaboration, code review, and code management for open source and private projects. Public projects are always free. “Github.com

GITHub is like facebook for developers. It’s a place where you can sync your local GIT repository to a central location, and then sync that central location to other local repositories.

Different people can connect to the same repository allowing multiple people to work on the same project.

What’s a repository?

A repository is essentially a collection of files that make up a project. You could think of it like a folder or directory. That analogy is not exact as it’s possible for a repository to have multiple sub-folders or directories, but it’s close enough for our purposes.

Is GIT only for Code?

GIT was definitely designed for software developers to as a versioning control system while developing software, but you can use it for tracking changes to things other than

You could use it for anything text format that you want to track changes to over time. For example

  • grocery lists
  • contact list
  • tracking your weight

There are a lot of interesting uses for GIT, one of those that we’re going to use today is looking at storing our Jinja2 templates on a public GIT repository and loading them directly into our python script as part of the code.

 

Import Required Libraries

Unles you’ve already got them, you’ll need to  pip install jinj2  and  pip install requests these two libraries before loading them into your running environment.

In [1]:
import requests
import yaml
import githubuser
from jinja2 import Environment, FileSystemLoader, Template
 

Loading Templates from GITHub

Like with most things in python, if it’s useful enough, chances are there’s probably someone else who already put a library together for that. In our case, we’re going to use the python request library to handle loading files directly from our Github repository.

 

The first thing we’ll do is load the HPE comware switch template from that we used in this post. If you wanted to take a look at this directly on github, it can be found here. All we have to do is to copy and paste the URL from our browser directly into the first input of the requests.get function.

note: The requests function will return a whole object that has various attributes. the ” .text ” at the end of this tells the function to just give us the contents of the file, not of the other information, like the HTTP status_code.

Simple, right?

In [75]:
comware_template = requests.get('https://github.com/netmanchris/Jinja2-Network-Configurations-Scripts/blob/master/simple_comware.j2').text
 

Looking at the output

So now that we’ve loaded the contents of the simple_comware.j2 template directly from the Github site into the comware_template variable. Let’s take a look to make sure that we have what we need.

In [76]:
print (comware_template)
 
<!DOCTYPE html>
<html lang="en" class="">
  <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# object: http://ogp.me/ns/object# article: http://ogp.me/ns/article# profile: http://ogp.me/ns/profile#">
    <meta charset='utf-8'>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta http-equiv="Content-Language" content="en">
    <meta name="viewport" content="width=1020">
    
    
    <title>Jinja2-Network-Configurations-Scripts/simple_comware.j2 at master · netmanchris/Jinja2-Network-Configurations-Scripts · GitHub</title>
    <link rel="search" type="application/opensearchdescription+xml" href="/opensearch.xml" title="GitHub">
    <link rel="fluid-icon" href="https://github.com/fluidicon.png" title="GitHub">
    <link rel="apple-touch-icon" href="/apple-touch-icon.png">
   
...
 

Hmmmmm. That’s not right?

The requests library is reaching out and grabbing whatever we put into that first variable. If we look at the print contents, we can see the first line is<!DOCTYPE html> . So it looks like we’re grabbing the rendered webpage, not just the contents of the file. Thankfully, looking at the GITHub website, there’s an option to look at any of your files in raw mode. So let’s grab that URL and try this again, ok?

In [77]:
comware_template = requests.get('https://raw.githubusercontent.com/netmanchris/Jinja2-Network-Configurations-Scripts/master/simple_comware.j2').text
In [78]:
print (comware_template)
 
#sysname config
sysname {{ simple['hostname'] }}
#vlan config
{% for vlan in simple['vlans'] -%}
vlan {{ vlan['id'] }}
    name {{ vlan['name'] }}
    description {{ vlan['description'] }}
{% endfor %}#snmp_config
snmp-agent
snmp-agent community read {{ simple['snmp']['read'] }}
snmp-agent community write {{ simple['snmp']['write'] }}
snmp-agent sys-info contact {{ simple['snmp']['syscontact']  }}
snmp-agent sys-info location {{ simple['snmp']['syslocation'] }}
snmp-agent sys-info version all
 

Ahhhh… That’s better.

 

Loading Network Specific Values from GITHub

Now we’re going to load our network specific values which were stored in the YAML file in this post. But this time, we’re going to load them directly from a private github repository.

The free GITHub accounts allow you to have public repositories, which means everyone can see what you’re doing, but if you have a paid version, you can get private repositories for as little as five dollars a month.

The private repositories are secured and can only be accessed by someone with a GIThub username and password who has explicitly been given access to this repository.

I would say that it’s probably a bad idea for us to keep any secure information like usernames, passwords, or SNMP strings in a online repository. But for my purposes, I don’t have anythng of value in this lab environment so I’m not too worried about it.

note: Before you put any sensitive data into an online repository of any kind, be sure to check with your companies data policies to see if you’re breaking any corporate rules.

 

Creating an Auth Object

First, I’m going to create an auth object, which is basically a single object that represents the username and password for my github account. In my case, I’ve got a file on my local hard drive that will automatically create the auth object for me.

In case you’re interested, the file is called githubuser.py and contains the following code. 

 

from requests.auth import HTTPBasicAuth

def gitcreds(): auth = HTTPBasicAuth('netmanchris', 'my_secret_password') return auth

In [79]:
auth = githubuser.gitcreds() #you didn't think I was going to give you my password did you?
 

Loading simple.yaml

We’ll now load the simple.yaml file like we did in this post but instead of opening it from a local file, we’re going to load it directly from the raw version of the file on github. I’d give you the link but it’s in a private repository, so you won’t be able to access it anyways.

Thigs I want to point out

  • yaml.load: takes the response and processes the yaml content directly into a python data structure ( dictionary )
  • .text: takes the “.text” attribute from the requests object which is the content of the page.
  • auth = auth: takes the auth object we created above and passes it as the username and password during the HTTP request.

Make sense?

In [80]:
simple = yaml.load(requests.get('https://raw.githubusercontent.com/netmanchris/PrivateRepo/master/simple_config.yaml', auth=auth).text)
In [81]:
simple
Out[81]:
{'hostname': 'testswitch',
 'ip': '10.101.0.221',
 'snmp': {'read': 'supersecret',
  'syscontact': 'admin.lab.local',
  'syslocation': 'lab',
  'trap': [{'target': '10.101.0.200'},
   {'target': '10.101.0.201'},
   {'target': '10.101.0.202'}],
  'write': 'macdonald'},
 'vlans': [{'description': 'management vlan',
   'id': '10',
   'name': 'management'},
  {'description': 'users vlan', 'id': '15', 'name': 'users'},
  {'description': 'phones vlan', 'id': '16', 'name': 'phones'},
  {'description': 'servers vlan', 'id': '20', 'name': 'servers vlan'}]}
 

Putting it all together

So looking at our list

  • download simple_comware.j2 template from Github public repo: **Check!**
  • download simple.yaml values file from Github private repo: **Check!**
  • rendered templates: **Nope**

So I guess we know what comes next, right?

 

Rendering the final config

We use the Template function to create a jinja2 template object and then we use the simple variable we created during the yaml section as input into the cw_template object.

In [82]:
cw_template = Template(comware_template)
type(cw_template)
Out[82]:
jinja2.environment.Template
In [83]:
print (cw_template.render(simple=simple))
 
#sysname config
sysname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp_config
snmp-agent
snmp-agent community read supersecret
snmp-agent community write macdonald
snmp-agent sys-info contact admin.lab.local
snmp-agent sys-info location lab
snmp-agent sys-info version all
 

Writing the Config to Disk

So far we’ve only been rendering and printing configurations, but it would be kinda nice to be able to have these on disk so that we can open them in our favorite editor before we cut and paste them into a telnet session to our network device.

The next two commands simply write the rendered template to disk with the filename comware.cfg and then we open and print the file to screen just to make sure it worked.

In [84]:
with open('comware.cfg', "w") as file:
    file.write(cw_template.render(simple=simple))
In [85]:
with open('comware.cfg') as file:
    print (file.read())
 
#sysname config
sysname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp_config
snmp-agent
snmp-agent community read supersecret
snmp-agent community write macdonald
snmp-agent sys-info contact admin.lab.local
snmp-agent sys-info location lab
snmp-agent sys-info version all
 

What’s next?

So far, we’ve come pretty far. We’ve written a couple of jinja templates, we’ve figure out how to store those files in a centralized control versioning system, but we’re still cut’ing and past’ing those configurations ourselves which is not ideal.

In the next post, we’ll look at using APIs to push the configuraiton directly to a configuraiton management tool.

Questions or comments? Feel free to post below!

@netmanchris

More Jinja – Working with YAM as an Input

 

Jinja2 Simple YAML Example


We’re now going to take a look at grabbing a file from the hard drive written in YAML syntax. YAML is arguably the most human readable data serialization format which makes it really easy for coders and non-coders alike to work with.

We’re going to build on the last Jinja2 example. Instead of creating the templates and variables directly in python. We’re going to load them instead from files on our computer.

This may seem like a small detail, but this allows us to deconstruct the building of our configurations, meaning that different people can be responsible for different components of the configuration. As with anything, if you can break a complex process down into several smalller less complex tasks, the whole thing starts to feel easier.

Loading Libraries

We’ll start by loading the required libraries

In [2]:
import yaml
from jinja2 import Environment, FileSystemLoader, Template
 

Set the Environment

Essentially, this set’s the path which will define the directory where the templates will be loaded from. In this case, I’m setting it to load from the same directory.

In [3]:
ENV = Environment(loader=FileSystemLoader('./'))
 

Printing the YAML File

In this step we’re going to use the with open() as file: way of opening the file and just printing it out so you can have a look at what the YAML syntax actually looks like. The advantage of using the with open as… way of opening the file is that there’s no need to go back and close the file when you’re done. May not sound like a big deal, but trust me, files don’t like to be orphaned in an open pyton variable somewhere.

If you’re comfortable with configuring a network device by hand, you can PROBABLY figure out exactly what these variables are doing. The hostname istestswitch, the SNMP configuration are in the snmp section. The read string is supersecret, the vlans are in the vlans section, etc…

Really easy to understand exactly what’s going on here, right?

In [15]:
with open('simple_config.yaml') as file:
    print (file.read())
 
hostname: testswitch

ip: 10.101.0.221


snmp:
    read: supersecret
    write: macdonald
    syscontact: admin.lab.local
    syslocation: lab
    trap:
    - {target: 10.101.0.200}
    - {target: 10.101.0.201}
    - {target: 10.101.0.202}



vlans:
- {description: management vlan, id: '10', name: management}
- {description: users vlan, id: '15', name: users}
- {description: phones vlan, id: '16', name: phones}
- {description: servers vlan, id: '20', name: servers vlan }
 

Loading the YAML File

In this next step, we’ll use the same with open as… way of opening the file, except rather than just reading the file this time, we’re going to use the yaml.loadmethod to load the contents of that file into a python variable called simple.

The nice thing is that the YAML library takes care of all the hard work for us. Automation is supposed to make things easier, right?

In [12]:
with open("simple_config.yaml") as simple:
    simple =  yaml.load(simple)
 

Looking at the simple variable

So now that we’ve loaded the contents of simple.yaml into the simple variable, let’s take a look at what python sees. We’ll run the type() command first to see what kind of object simple has been created as, and then we’ll print out the contents of simple so you can see how the YAML library has transformed it from the text in the YAML file to something which is easier to work with in python.

In [16]:
type(simple)
Out[16]:
dict
In [11]:
simple
Out[11]:
{'hostname': 'testswitch',
 'ip': '10.101.0.221',
 'snmp': {'read': 'supersecret',
  'syscontact': 'admin.lab.local',
  'syslocation': 'lab',
  'trap': [{'target': '10.101.0.200'},
   {'target': '10.101.0.201'},
   {'target': '10.101.0.202'}],
  'write': 'macdonald'},
 'vlans': [{'description': 'management vlan',
   'id': '10',
   'name': 'management'},
  {'description': 'users vlan', 'id': '15', 'name': 'users'},
  {'description': 'phones vlan', 'id': '16', 'name': 'phones'},
  {'description': 'servers vlan', 'id': '20', 'name': 'servers vlan'}]}
 

Templates


Printing the simple_cisco template

In this step we’re going to take a look at the simple_cisco.j2 template that I’ve created. This is a very simple configuration just to show the power of jinja for making your network configurations.

If you look closely, you’ll see a couple of {% for…%} tags in here. This is a control structure called an interator, commonly known as a For loop. Essentially it’s saying, take the list of things and do this one action to each of the things in the list.

One thing to note here is that j2 is not a jinja2 specific file extention, but just something that I, and many others I’m sure, use to designate their template files.

In [28]:
with open('simple_cisco.j2') as file:
    print (file.read())
 
#hostname config
hostname {{ simple['hostname'] }}
#vlan config
{% for vlan in simple['vlans'] -%}
vlan {{ vlan['id'] }}
    name {{ vlan['name'] }}
    description {{ vlan['description'] }}
{% endfor %}#snmp config
snmp-server community {{ simple['snmp']['read'] }} RO
snmp-server community {{ simple['snmp']['write'] }} RW
snmp-server ifindex persist
snmp-server location {{ simple['snmp']['syslocation'] }}
snmp-server contact {{ simple['snmp']['syscontact'] }}
{% for trap in simple['snmp']['trap'] -%}
snmp-server host {{ trap['target'] }}  public
{% endfor %}
 

Rendering the simple_cisco template

In this last step of working with the simple_cisco template, we’re going to now pass this through the jinja rendering engine. Because we’ve got dynamic parts in the template, we’re going to have to supply a source for the variables to fill the dynamic part in. If you look below, we’re saying that anytime you see the wordsimple in the template, you should look in the variable simple we created above and see if there’s the appropriate information there to fill it in.

In simpler terms, we’re going to take the simple.yaml file we loaded above as the input values into this template.

We then render the template and…

In [29]:
template = ENV.get_template("simple_cisco.j2")
print (template.render(simple=simple))
 
#hostname config
hostname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp config
snmp-server community supersecret RO
snmp-server community macdonald RW
snmp-server ifindex persist
snmp-server location lab
snmp-server contact admin.lab.local
snmp-server host 10.101.0.200  public
snmp-server host 10.101.0.201  public
snmp-server host 10.101.0.202  public

 

Look familiar right? Minimal typing. Alll the VLANS are there, etc… and the best thing is if I run this a thousand times, it will always come out the same way.

 

But I have a multi-vendor network!!!!

This is the true power of jinja for me. I happen to run multiple vendors in my lab, but I’d like to have the ability to drive all of the configurations from a central location to make sure that they all have the same vlans, usernames and passwords, snmp strings, etc…

So now let’s take a look at running another vendor’s template using the same simple.yaml file as the input source.

 

Printing the simple_comware template

In this step we’re going to take a look at the simple_comware.j2 template that I’ve created. This is a very simple configuration just to show the power of jinja for making your network configurations.

You’ll notice that it’s very close to the simple_cisco.j2 file shown above. The real difference is the parts outside of the jinja2 variables. HPE Networking’s comware devices use the keyword sysname instead of hostname. They use the keyword snmp-agent instead of snmp-server. Minor differences in the syntax, but the actual values are exactly the same for both devices.

In [30]:
with open('simple_comware.j2') as file:
    print (file.read())
 
#sysname config
sysname {{ simple['hostname'] }}
#vlan config
{% for vlan in simple['vlans'] -%}
vlan {{ vlan['id'] }}
    name {{ vlan['name'] }}
    description {{ vlan['description'] }}
{% endfor %}#snmp_config
snmp-agent
snmp-agent community read {{ simple['snmp']['read'] }}
snmp-agent community write {{ simple['snmp']['write'] }}
snmp-agent sys-info contact {{ simple['snmp']['syscontact']  }}
snmp-agent sys-info location {{ simple['snmp']['syslocation'] }}
snmp-agent sys-info version all
 

Rendering the simple_comware template

In this step, we’re going to render and print the template using the same simple variable that we used for the simple_cisco.j2 template above. Because of the difference in the simple_comware.j2 templates, it will render with the proper syntax for the HPE devices.

In [31]:
template = ENV.get_template("simple_comware.j2")
print (template.render(simple=simple))
 
#sysname config
sysname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp_config
snmp-agent
snmp-agent community read supersecret
snmp-agent community write macdonald
snmp-agent sys-info contact admin.lab.local
snmp-agent sys-info location lab
snmp-agent sys-info version all
 

Wrap Up

I’m sure you’ll agree this is a much better way of creating your configurations than grabbing a console cable and typing them all out by hand, right? But there’s still a lot of room for improvement! Currently, we’re going to render the template and then cut and paste them into a console cable. So although we’ve made some gains… it could be better.

We’ll take a look at that next time.

@netmanchris

 

Intro to Jinja2

 
 

What is Jinja2

Jinja2 is a templating language that was originally used as part of the Flask python web framework. From the Jinja2 website

Jinja2 is a full featured template engine for Python. It has full unicode support, an optional integrated sandboxed execution environment, widely used and BSD licensed

It was originally developed to help automatically generate HTML dynamically as part of the flask framework, more on that in another post, but it can also easily be used to help us generate our configuration files for our infrastructure devices.

This is going to be a very simple introduction to a few of the basic concepts of that jinja uses which, hopefully, will help to understand how Jinja can be used as a first step down the road of gaining automation skills.

We’ll take a look at a developing some intuition on how Jinja2 can be used to create basic network infrastructure device configurations. This is definitly not the modern method of interfacing directly into the control/data/management plane of devices using APIs, but it’s definitely a step in the right direction on understanding how a bit of code can help make your life better.

Prereqs

I’m assuming you’ve already got python installed on your system. You’re also going to need to run the pip install jinja2 command from a terminal window to get the latest version of jinja2 which should work just fine here.

 

Learning by Example

In this section we’ll start with a small example on how to create a few VLANs using the typical syntax from a modern networking OS. In this case, I used the HPE Comware syntax, but it would be easy enough to create this using a Cisco or Juniper configuration and you’re encouraged to try to get this working with your own network vendor.

 

Import required libraries

First We’ll import the required modules from the Jinja2 library. This is pretty much stolen directly from the jinja2 docs.

In [1]:
from jinja2 import Environment, FileSystemLoader, Template
 

Creating the VLANS

For this example, we’re going to create a python list of dicts which contains six different VLANS as listed in the tabel below.

A python dictionary is just a key-value pair, where the value for a specific key in the dictionary can be accessed using the key name.

  • Name: Name of the VLAN
  • Description: Descrition of the VLAN
  • VLAN ID: Dot1q VLAN ID.
Name Description VLAN ID
Management Management VLAN 10
Users Users VLAN 15
Phones Phones VLAN 16
Servers Servers VLAN 20
Mobility Mobility VLAN 30
Guest Guest VLAN 40
In [5]:
vlans_list = [{'name': 'management', 'description': 'management vlan', 'id': '10'},
         {'name': 'users', 'description': 'users vlan', 'id': '15'},
         {'name': 'phones', 'description': 'phones vlan', 'id': '16'},
         {'name': 'servers', 'description': 'servers vlan', 'id': '20'},
         {'name': 'mobility', 'description': 'mobility vlan', 'id': '30'},
         {'name': 'guest', 'description': 'guest vlan', 'id': '40'},
         {'name': 'rob', 'description': 'guest vlan', 'id': '45'}
         ]
 

VLAN Jinja2 Template

In this step, we’re going to create a variable called text_file which will contain the content of a jinja2 template. This is a basic python string object which means, at this point, it’s just a bunch of text.

In normal circunstances, we would actually be reading this template from a file located on the hard drive, but for our purposes today, we’ll just put the templatein by hand.

What makes Jinja2 powerful is the control structures that allow it to perform programatic operations. In this example, we’re creating a For loop.

Following the code we will each vlan in the vlans object we created above and then render the template using the ‘id’ key for the first variable, the ‘name’ key for the second variable, and the ‘description’ key for the last variable.

Hopfully, this makes sense, but if not, just hold on and it should become clear before the end.

In [6]:
text_file = ('''
#vlan config
{% for vlan in vlans -%}
vlan {{ vlan['id'] }}
 name {{ vlan['name'] }}
 description {{ vlan['description'] }}
{% endfor %}''')
 

If I was to write the same as a traditional python iterator it would look something like this. You can see how they are related I hope?

In [7]:
for vlan in vlans_list:
    print ('''vlan ''' +vlan['id']+
           '''\n name '''+vlan['name']+
           '''\n description '''+vlan['description'])
 
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers
    description servers vlan
vlan 30
    name mobility
    description mobility vlan
vlan 40
    name guest
    description guest vlan
vlan 45
    name rob
    description guest vlan
 

That’s a lot of work typing isn’t it?

You could ask

That’s more typing than I would do by hand? Why would I use this?

Great question. The point of automating anything is to cut down on the repetitive effort it takes to accomplish a given goal. In this case, we can simply count the number of key strokes it would take to create a single new VLAN on a switch.

In [6]:
count_chars = "vlan', 'name', 'description"
keystrokes = len(count_chars)
print (keystrokes)
 
27
 

Now let’s pretend we had to type that 10 times

In [7]:
keystrokes * 10
Out[7]:
270
 

Or maybe we had to create 100 VLANs.

In [10]:
keystrokes * 100
Out[10]:
2700

Ir maybe we had to create the full 4094 VLANs available

In [11]:
keystrokes * 4094
Out[11]:
110538
 

Not sure about you, but if I don’t have to type 110,000 keystrokes, my fingers will love me at the end of the day. Not to mention the fact that it’s also repeated perfectly every single time, not a single typo in there.

 

Create the Template Object

Now that we’ve created the text_file string object, we need to transform it into a jinj2 template which will allow us to then render it. We will create a new object called vlan_template and assign an instance of the Template class using the text_file contents as the input.

In [12]:
vlan_template = Template(text_file)
 

Make the Magic Happen

We will now use the render method on the vlan_template that we created above. We have a single argument to pass into the function. In this case we are passing the vlans_list list of dictionaries we create above in to the function as the vlans variable.

In [13]:
vlan_template.render(vlans=vlans_list)
Out[13]:
'\n#vlan config\nvlan 10\n    name management\n    description management vlan\nvlan 15\n    name users\n    description users vlan\nvlan 16\n    name phones\n    description phones vlan\nvlan 20\n    name servers\n    description servers vlan\nvlan 30\n    name mobility\n    description mobility vlan\nvlan 40\n    name guest\n    description guest vlan\n'
 

Hmmm What happened there?

That doesn’t look like a configuration file does it? The output of this file is actually a python string object. In python, we need someway to represent a carriage return (enter-key) and the \n just happens to have that honour.

Instead of running the template rendering directly, we can instead capture the output into a string object which we will then pass to the print command.

In [14]:
rendered_template = vlan_template.render(vlans=vlans_list)
In [15]:
print (rendered_template)
 
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers
    description servers vlan
vlan 30
    name mobility
    description mobility vlan
vlan 40
    name guest
    description guest vlan

 

Clear?

Hopefully, this has shown you a bit of how a basic jinja control structure, like a For loop, can be used to cut down on a lot of key strokes, increase the accuracy of the configurations and help to streamline the operations.

In the next post, I’ll look at loading YAML files directly into python and using their contents as input into some more advanced jinja2 templates.

 

I use SNMP SETs and I’m not afraid to admit it.

Do you remember back in CCNA school when we learned all sorts of great things that we very rarely followed. One of the favourites was that we are supposed to put meaningful descriptions on all of our interfaces so we know what the other side is connected to.

How many people actually follow that advice?

Yeah, I never do it either. There’s always just too many things on the list that need to get done and it seems like that extra 5 seconds it would take me to update the description to the interface just doesn’t seem like it’s worth the effort. Of course, then I later check the port and end up knocking out my XYZ services and cause myself an outage.

This is where a little python and a decent NMS can help to solve a problem.

Understanding ifIndex

Before we get into the code. We need to understand a little about ifIndex values and how they relate to the physical interfaces of the devices. If you’re REALLY interested, you can do some reading in RFC 2863.  But in a nutshell, each interface on a device, whether physical or logical has a specific numeric value assigned to it which is the last digit in the interface statistics that can be seen through the SNMP interfaces. This is commonly known as interfaces group stats.

There are a bunch of different tables in the interface group stats that are used to store specific kinds of information or statistics for the interfaces such as

  • ifNumber (.1.3.6.1.2.1.2.1.0) – Which is the total number of Interfaces
  • ifIndex ( .1.3.6.1.2.1.2.2.1.1. * ) – Which acts as a primary key for all the other interface group stats.
  • ifDescr (.1.3.6.1.2.1.2.2.1.2.* ) – Which is the name of the interface
  • ifType (.1.3.6.1.2.1.2.2.1.3.* ) – Which describes the type of the interface
  • ifMTU ( .1.3.6.1.2.1.2.2.1.4.*) – Which shows the current MTU size configured on the interface
  • ifSpeed (.1.3.6.1.2.1.2.2.1.5.*) – Which shows the current speed of the interface
  • ifPhysicalAddress (.1.3.6.1.2.1.2.2.1.6.*) – Which shows the mac address of the interface
  • ifAdminStatus (.1.3.6.1.2.1.2.2.1.7.*) – Which shows the current admin status of the port
  • ifOperStatus (.1.3.6.1.2.1.2.2.1.8.*) – Which shows the current operational status

There’s a bunch more which you can see here if you’re interested, but one that I found particularly fun was the ifAlias ( .1.3.6.1.2.1.31.1.1.1.18.* ) which actually corresponds to the description command in your friendly neighbourhood network operating system

So when your interface is configured like this

Screen Shot 2015 12 10 at 10 33 12 PM

The ifAlias value for the corresponding ifIndex looks like this

Screen Shot 2015 12 10 at 10 34 30 PM

The Fun Begins

The interesting part about the ifAlias value is that it’s actually a SET-able value through the SNMP interface. That means that if you have a simple piece of python code like the following

It will allow you to run a little command like this >>> set_interface_description(‘10.101.0.221’, ‘1’, ‘Changed This’ ) which will result in the following.

Screen Shot 2015 12 10 at 10 52 41 PM

So this is cool, right? We’ve just programatically changed a single interface description on a interface. We could stop here and you would be left with a

“So what? Why would I go to the trouble of doing that. It’s harder than just typing in the description manually!”

But wait! There’s more…

The really cool part about automating something so simple is that now we have a building block that we can do something with. For those of you who might have seen the HPE Intelligent Management Centre, you may already be aware that the topology map does this really cool thing where it actually creates a table of all of the links within a given custom view and automatically creates link-names for them.

Screen Shot 2015 12 10 at 10 58 57 PM

You’ll notice that the auto-generated Link-Name actually tells me who’s connected to both sides of that link. You’ll also notice that I have the left and right nodes ( with the IP address ), as well as the left and right interfaces. And  “yes”, there is an API for this where it’s all represented in a nice little JSON string which can be easily parsed in your favourite IDE.

Time vs ROI

The reason we don’t label interfaces is that most people feel it’s just not worth the effort to keep them up to date. Things change too often and it’s just too easy to say “it’s not that important” and move on before changing that description.  I’m with you. I’m as guilty as everyone else on this. But with the help of a little code, a good NMS, the entire process is now automated.

I can proudly say that, for now at least, my lab is 100% accurate as far as interface descriptions go. And because the whole thing is totally automated, I simply re-run the script overtime I make some changes.

So for those of you who weren’t aware. Yes, there is someone who actually uses SNMP SETs in the world. 🙂

2015 Recap and plans for 2016

About this time last year, I wrote this post.  It’s time to revisit again and plan for 2016. 

 

How did I do?

In 2015, I planned to work on skills in four major areas; python, data science, virtualization, and then just keep up on networking.  In general, I think I did good in all areas, with the breakaway really being in the python area. I made a concerted effort this to year to seek out project after project that would allow me to explore different aspects of python, and force me to grow in as many areas as possible. Attending Interop sessions led by such trailblazers as Jason Edelman, Jeremy Schulman, and Matt Oswalt definitely gave me ideas and inspiration to explore new areas, push boundaries, and have really helped me to grow as a coder/developer not to mention really helped me to cement some opinions on the future of networking in general.

I did manage to get through a bunch of the R courses in Cousera and they were great. I’d love to say that I’m going to return and finish the specialization  (I’m two courses shorts) but if I”m honest, I’m probably going to move towards the Data Science aspects of Python and get into Anaconda and Pandas more in 2016. Nothing like combining two growth areas into one to really push the growth. 

 

More so this year that others, having great conversations over beers, Moscow Mules, and many a sugery umbrella drink helped me expand my knowledge and firm up some of the thoughts I’ve been having for the last couple of years. No need to name drop, but you all know who you are and I’d like to say thanks for all of the conversations and laughs. As much as I love the tech, it’s the people that always help to drive me forward. 

If I’m keeping score, I think that 2015 was a good year. 

 

Plans Plans Plans

Now comes time to publicly declare what I want to accomplish in 2016. This is always the scary part as I know I’m now publicly accountable for any grandiose designs I throw into the ether. 🙂 

 

Practice to Application:

 

I’ve got a bit of a lab at home. If things go as planned, at the end of the year, I will be able to factory reset *almost* the entire lab and have it come back from the dead in a completely automated fashion. The plan here is to use a combination of python, jinja2, IMC, Ansible, and whatever other pieces I need to duck tape together to make this work by the end of the year.  Just because I like to make my life harder than it has to be, I’m planning on building out the topology using vendor independent methodologies, meaning that I want to be able to place a Cisco, HP, or Juniper box into any position in my lab access/distribution/core/dmz/wan/etc… and use a YAML file to dynamically build the required configurations on demand.  

 

Yeah… I know….   But it’s good to set goals right?

 

OpenSwitch

OpenSwitch is also another area which I will definitely be exploring during 2016.  The project is still very new and definitely has some places, mostly in the documentation area, where there’s room to make a difference. I’ve been really lucky to be able to work at a place where I have direct access to some of the projects core developers and I’m hoping I can share the fruits of that access in more blogs posts, pull requests to enhance the documentation, as well as some interoperability testing with some of the usual-suspect network kit that I already have in my lab. Right now, I’m thinking OSPF, BGP, Spanning-Tree as an unambitious start, and moving from there to using the declarative interface and REST interfaces to see how I can incorporate it into project one.

 

 

Thoughts on 2015

2015 was a good year. As an industry, I think we’ve made some great gains in general. The whole “Is SDN really a thing?” conversations seem to be over and we’ve moved on to “ I don’t care what you call it, what does it do for me?” conversations  are starting to really get interesting.  The projects with value are starting to separate themselves from the science fair exhibits and it looks like parts of the networking profession are finally past the out-right denial and have reached the bargaining stage ( “ Can someone else write the scripts for me? Please? ) 

I’ve been able to make forward momentum in all the areas I wanted to and I’m generally where I thought I was going to be at the start of the year.

 

Looking forward to 2016!

 

@netmanchris

Installing OpenSwitch

 

First, disclaimer: I’m an HP employee. HP’s a major contributor to the OpenSwitch project. Just thought you should know in case you think that affects my opinion here.

If you need more info on the OpenSwitch project, you can check out the first post in this series here.

Getting our hands dirty

This section comes down into three steps, if you don’t follow these steps, you won’t succeed. I’m not going to go into details on these steps as I’m assuming you can figure this out if you found this blog. 🙂 

– Install Virtual Box

– Install Vagrant

– Install Docker Toolbox

I’m running OpenSwitch on a Windows box in this case as the documentation covers the ‘IX build. I’m running on this natively on OSX which also means that I’ve got to install the docker toolbox plugin to get docker containesr to work. I’m  also assuming that you’ve already installed Virtual Box and Vagrant for the following section.

Installing Vagrant Plugin

From a terminal window, run the vagrant plugin install vagrant-reload command from the CLI. This should show the following output.

Screen Shot 2015 10 20 at 8 28 45 PM

Installing the OpenSwitch Dev Environment

For this section, I’m assuming you have already downloaded the vagrant files from here into your working directory.

Running the Docker Toolbox Plugin

Run the Docker QuickStart Terminal application and wait for the virtual box image to come to a running state. You should be able to see the following 

Screen Shot 2015 10 20 at 8 30 50 PM

Vagrant up!

From the terminal, navigate to where you have unzipped the OpenSwitch vagrant files that you downloaded from here.  Run vagrant up command from the CLI. At this point some magic happens ( read more on Vagrant here if you’ve never worked with this tool before. Magic is obviously not magic, but I just don’t feel like explaining the whole process in this post. )

 

On a OSX box, you’re not running as root so you may end up with the following window

Screen Shot 2015 10 20 at 8 33 27 PM

If you hit this, don’t worry, just SUDO it! 

Screen Shot 2015 10 20 at 8 36 39 PM

 Accessing the OpenSwitch

From the same terminal window issue the sudo vagrant ssh command to be able to access the shell (CLI) of the OpenSwitch. 

 

If you are successful, you should see the following output. Notice the shell has changed to vagrant@switch

Screen Shot 2015 10 20 at 8 39 21 PM

Accessing the network interface

From the vagrant@switch prompt, issue the sudo vtysh command and you will now have access to an industry standard hierarchical CLI like we all know and love!

Screen Shot 2015 10 20 at 8 41 36 PM

 My thoughts so far

Getting this up and running has been relatively painless. There were a couple of small things to get it running that were particular to OSX which was not covered on the OpenSwitch quick start guide. Nothing that a little patience and google didn’t help me cover in a few minutes though.  The install experience was pretty easy.  The guides were pretty accurate and the getting this up and running should be something most of us can follow without much trouble.

OpenSwitch doesn’t have what I would call a robust network stack at this point in time, but we’re still really early in this world.  Now that I’ve got it up and running, I’m looking forward to starting to look at the alternate interfaces such as OVSDB and REST as described here

Anyone else got this up and running yet? Thoughts? Let me know in the comments below!

 

@netmanchris