GIT and Jinja – Like Peanut butter and Pickles!


Thanks to @mierdin for point this out. It looks like the wordpress format is causing some strange word-wrap issues. For a better view please click here to see the full post without presentation issues. 

 

Using GITHub to build our Network Configs

As I wrote in this post, one of my goals for this year is to be able to compltely automate the build of my lab environment programatically.

In the last couple of jinja posts, I wrote about the basics of Jinja2 templates and how they can be applied to building network configurations.

In this post, I’m going to take the next step and move those files from my local hard drive out to…

 

duh duh dahhhhhhhhhh

The cloud.

The cloud

 

Before we get started…

We’re going to go over some basics on the tools we’re using to make sure everyone’s on the same page. cool?

What’s GIT?

Git is a widely-used source code management system for software development. It is a distributed revision control system with an emphasis on speed, data integrity, and support for distributed, non-linear workflows. wikipedia

Huh?

GIT is a piece of software that allows you to track changes to files over time.

So what’s GITHub?

“Where software is built Powerful collaboration, code review, and code management for open source and private projects. Public projects are always free. “Github.com

GITHub is like facebook for developers. It’s a place where you can sync your local GIT repository to a central location, and then sync that central location to other local repositories.

Different people can connect to the same repository allowing multiple people to work on the same project.

What’s a repository?

A repository is essentially a collection of files that make up a project. You could think of it like a folder or directory. That analogy is not exact as it’s possible for a repository to have multiple sub-folders or directories, but it’s close enough for our purposes.

Is GIT only for Code?

GIT was definitely designed for software developers to as a versioning control system while developing software, but you can use it for tracking changes to things other than

You could use it for anything text format that you want to track changes to over time. For example

  • grocery lists
  • contact list
  • tracking your weight

There are a lot of interesting uses for GIT, one of those that we’re going to use today is looking at storing our Jinja2 templates on a public GIT repository and loading them directly into our python script as part of the code.

 

Import Required Libraries

Unles you’ve already got them, you’ll need to  pip install jinj2  and  pip install requests these two libraries before loading them into your running environment.

In [1]:
import requests
import yaml
import githubuser
from jinja2 import Environment, FileSystemLoader, Template
 

Loading Templates from GITHub

Like with most things in python, if it’s useful enough, chances are there’s probably someone else who already put a library together for that. In our case, we’re going to use the python request library to handle loading files directly from our Github repository.

 

The first thing we’ll do is load the HPE comware switch template from that we used in this post. If you wanted to take a look at this directly on github, it can be found here. All we have to do is to copy and paste the URL from our browser directly into the first input of the requests.get function.

note: The requests function will return a whole object that has various attributes. the ” .text ” at the end of this tells the function to just give us the contents of the file, not of the other information, like the HTTP status_code.

Simple, right?

In [75]:
comware_template = requests.get('https://github.com/netmanchris/Jinja2-Network-Configurations-Scripts/blob/master/simple_comware.j2').text
 

Looking at the output

So now that we’ve loaded the contents of the simple_comware.j2 template directly from the Github site into the comware_template variable. Let’s take a look to make sure that we have what we need.

In [76]:
print (comware_template)
 
<!DOCTYPE html>
<html lang="en" class="">
  <head prefix="og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# object: http://ogp.me/ns/object# article: http://ogp.me/ns/article# profile: http://ogp.me/ns/profile#">
    <meta charset='utf-8'>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta http-equiv="Content-Language" content="en">
    <meta name="viewport" content="width=1020">
    
    
    <title>Jinja2-Network-Configurations-Scripts/simple_comware.j2 at master · netmanchris/Jinja2-Network-Configurations-Scripts · GitHub</title>
    <link rel="search" type="application/opensearchdescription+xml" href="/opensearch.xml" title="GitHub">
    <link rel="fluid-icon" href="https://github.com/fluidicon.png" title="GitHub">
    <link rel="apple-touch-icon" href="/apple-touch-icon.png">
   
...
 

Hmmmmm. That’s not right?

The requests library is reaching out and grabbing whatever we put into that first variable. If we look at the print contents, we can see the first line is<!DOCTYPE html> . So it looks like we’re grabbing the rendered webpage, not just the contents of the file. Thankfully, looking at the GITHub website, there’s an option to look at any of your files in raw mode. So let’s grab that URL and try this again, ok?

In [77]:
comware_template = requests.get('https://raw.githubusercontent.com/netmanchris/Jinja2-Network-Configurations-Scripts/master/simple_comware.j2').text
In [78]:
print (comware_template)
 
#sysname config
sysname {{ simple['hostname'] }}
#vlan config
{% for vlan in simple['vlans'] -%}
vlan {{ vlan['id'] }}
    name {{ vlan['name'] }}
    description {{ vlan['description'] }}
{% endfor %}#snmp_config
snmp-agent
snmp-agent community read {{ simple['snmp']['read'] }}
snmp-agent community write {{ simple['snmp']['write'] }}
snmp-agent sys-info contact {{ simple['snmp']['syscontact']  }}
snmp-agent sys-info location {{ simple['snmp']['syslocation'] }}
snmp-agent sys-info version all
 

Ahhhh… That’s better.

 

Loading Network Specific Values from GITHub

Now we’re going to load our network specific values which were stored in the YAML file in this post. But this time, we’re going to load them directly from a private github repository.

The free GITHub accounts allow you to have public repositories, which means everyone can see what you’re doing, but if you have a paid version, you can get private repositories for as little as five dollars a month.

The private repositories are secured and can only be accessed by someone with a GIThub username and password who has explicitly been given access to this repository.

I would say that it’s probably a bad idea for us to keep any secure information like usernames, passwords, or SNMP strings in a online repository. But for my purposes, I don’t have anythng of value in this lab environment so I’m not too worried about it.

note: Before you put any sensitive data into an online repository of any kind, be sure to check with your companies data policies to see if you’re breaking any corporate rules.

 

Creating an Auth Object

First, I’m going to create an auth object, which is basically a single object that represents the username and password for my github account. In my case, I’ve got a file on my local hard drive that will automatically create the auth object for me.

In case you’re interested, the file is called githubuser.py and contains the following code. 

 

from requests.auth import HTTPBasicAuth

def gitcreds(): auth = HTTPBasicAuth('netmanchris', 'my_secret_password') return auth

In [79]:
auth = githubuser.gitcreds() #you didn't think I was going to give you my password did you?
 

Loading simple.yaml

We’ll now load the simple.yaml file like we did in this post but instead of opening it from a local file, we’re going to load it directly from the raw version of the file on github. I’d give you the link but it’s in a private repository, so you won’t be able to access it anyways.

Thigs I want to point out

  • yaml.load: takes the response and processes the yaml content directly into a python data structure ( dictionary )
  • .text: takes the “.text” attribute from the requests object which is the content of the page.
  • auth = auth: takes the auth object we created above and passes it as the username and password during the HTTP request.

Make sense?

In [80]:
simple = yaml.load(requests.get('https://raw.githubusercontent.com/netmanchris/PrivateRepo/master/simple_config.yaml', auth=auth).text)
In [81]:
simple
Out[81]:
{'hostname': 'testswitch',
 'ip': '10.101.0.221',
 'snmp': {'read': 'supersecret',
  'syscontact': 'admin.lab.local',
  'syslocation': 'lab',
  'trap': [{'target': '10.101.0.200'},
   {'target': '10.101.0.201'},
   {'target': '10.101.0.202'}],
  'write': 'macdonald'},
 'vlans': [{'description': 'management vlan',
   'id': '10',
   'name': 'management'},
  {'description': 'users vlan', 'id': '15', 'name': 'users'},
  {'description': 'phones vlan', 'id': '16', 'name': 'phones'},
  {'description': 'servers vlan', 'id': '20', 'name': 'servers vlan'}]}
 

Putting it all together

So looking at our list

  • download simple_comware.j2 template from Github public repo: **Check!**
  • download simple.yaml values file from Github private repo: **Check!**
  • rendered templates: **Nope**

So I guess we know what comes next, right?

 

Rendering the final config

We use the Template function to create a jinja2 template object and then we use the simple variable we created during the yaml section as input into the cw_template object.

In [82]:
cw_template = Template(comware_template)
type(cw_template)
Out[82]:
jinja2.environment.Template
In [83]:
print (cw_template.render(simple=simple))
 
#sysname config
sysname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp_config
snmp-agent
snmp-agent community read supersecret
snmp-agent community write macdonald
snmp-agent sys-info contact admin.lab.local
snmp-agent sys-info location lab
snmp-agent sys-info version all
 

Writing the Config to Disk

So far we’ve only been rendering and printing configurations, but it would be kinda nice to be able to have these on disk so that we can open them in our favorite editor before we cut and paste them into a telnet session to our network device.

The next two commands simply write the rendered template to disk with the filename comware.cfg and then we open and print the file to screen just to make sure it worked.

In [84]:
with open('comware.cfg', "w") as file:
    file.write(cw_template.render(simple=simple))
In [85]:
with open('comware.cfg') as file:
    print (file.read())
 
#sysname config
sysname testswitch
#vlan config
vlan 10
    name management
    description management vlan
vlan 15
    name users
    description users vlan
vlan 16
    name phones
    description phones vlan
vlan 20
    name servers vlan
    description servers vlan
#snmp_config
snmp-agent
snmp-agent community read supersecret
snmp-agent community write macdonald
snmp-agent sys-info contact admin.lab.local
snmp-agent sys-info location lab
snmp-agent sys-info version all
 

What’s next?

So far, we’ve come pretty far. We’ve written a couple of jinja templates, we’ve figure out how to store those files in a centralized control versioning system, but we’re still cut’ing and past’ing those configurations ourselves which is not ideal.

In the next post, we’ll look at using APIs to push the configuraiton directly to a configuraiton management tool.

Questions or comments? Feel free to post below!

@netmanchris

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s