Using JSONSchema to Validate input

There are a lot of REST APIs out there. Quite a few of them use JSON as the data structure which allows us to get data in and out of these devices. There are a lot of network focused blogs that detail how to send and receive data in and out of these devices, but I wasn’t able to find anything that specifically talked about validating the input and output of the data to make sure we’re sending and receiving the expected information.

Testing is a crucial, and IMO too often overlooked, part of the Infrastructure as Code movement. Hopefully this post will help others start to think more about validating input and output of these APIs, or at the very least, spend just a little more time thinking about testing your API interactions before you decide to automate the massive explosion of your infrastructure with a poorly tested script. 🙂

What is JSONSchema

I’m assuming that you already know what JSON is, so let’s skip directly to talking about JsonSchema. This is a pythonlibrary which allows you to take your input/output  and verify it against a known schema which defined the data types you’re expecting to see.

For example, consider this snippet of a schema which defines what a valid VLAN object looks like

"vlan_id":
{
    "description": "The unique ID of the VLAN. IEEE 802.1Q VLAN identifier (VID)", 
    "minimum": 1, 
    "maximum": 4094, 
    "type": "integer", 
    "sql.not_null": true
}

You can see that this is a small set of rules that defines what is a valid entry for the vlan_id property of a VLAN.  As a network professional, it’s obvious to us that a valid VLAN ID  must be between 1 and 4094. We know this because we deal with it every day. But our software doesn’t know this. We have to teach it what a valid vlan_id property looks like and that’s what the schema does.

our software doesn’t know this. We have to teach it

Why do we care?

Testing is SUPER important. By being able to test the input/output of what you’re feeding into your automation/orchestration framework, it can help you to avoid, at worst, a total meltdown, or, at best, a couple of hours trying to figure out why your code doesn’t work.

Using JSONSchema

So the two things you’re going to need to use JSONSchema are

  • The JSON Schema for a specific API endpoint
  • The input/output that you want to validate.

In this case, we’ll use a VLAN object that is coming out of an ArubaOS-Switch REST API.

You did know the ArubaOS-Switches have a REST API, right?

Step1 – Loading the VLAN object

We’re going to gather the output from the VLANS API. Instead of writing custom code, we’ll just use the pyarubaoss library. I’ll leave you to check out the GitHub repo and just paste the contents of the output of a single VLAN JSON output here. I’m also going to create a second VLAN with a VLAN_ID of 5000. Just to show how this works. 5000 of course, is not valid and we’d like to prove that. Right?

Step 2 – Loading the JSON Schema Definition

Now we have the output, we want to make sure that the output here complies with the JSON Schema definition that we’ve been provided.

Loading the JSON schema

Here’s a sub-set of the JSON schema that defines what a valid VLAN looks like

Step 3 – Importing the JSON Schema Library and Validating

Now we’re going to load the JSON Schema library into our python session and use it to validate the VLAN object using the Schema we defined above.

First we’ll look at the vlan_good object and run it through the validate function

As you can see, there’s nothing to see here. Basically this means that the vlan_good object is conforming properly to the provided JSON Schema. The VLAN ID is valid as it’s a integer value between 1 and 4094

Now let’s take a look at the vlan_bad object and run it through the same validate function

We can see that the validate function now raises an exception because and let’s us know very specifically that the VLAN ID 5000 is not valid

jsonschema.exceptions.ValidationError: 5000is greater than the maximum of 4094

Pretty cool right? We can still definitely shoot ourselves in the foot, but at least we know the input/output data that we’re using for our API is valid. To me this is important for two reasons

  • I can validate that the data I’m trying to send to a given API conforms to what that API is expecting
  • I can validate that the vendor didn’t suddenly change their API which is going to break my code

Wrap Up

There are a lot of networking folk who have started to take on the new set of skills required to automate their infrastructure. One of the most crucial parts of this is testing and validating the code to ensure that you’re not just blowing up your network more efficiently. JSON Schema is one of the tools in your tool box that can help you do that.

Comments, questions? Let me know

@netmanchris

Advertisements

HP IMC’s New eAPI

Now that I got rum-pooh out of my system…  on to a slightly more technical post.

Not sure if anyone of you caught the recent announcements about the new eAPI from HP’s Intelligent Management center.  In a nutshell, this is a RESTful API which allows programatic access to almost all ( maybe all?) of the IMC functions through an HTTP(s) interface.

Now I’m not a programer at all, but I like to think I have a working knowledge of programing logic. At least enough to give a half-decent programmer enough information to get the job done.

So when I had a co-worker present me with a problem earlier this week, I thought “Hey, I wonder what this new eAPI can really do?”.  I did mention I’m not a programer right?  After this little exercise, I’m thinking I might just have to pick up some scripting skills this year. 🙂

So what’s the value of the eAPI? It functionally allows IMC to act as the progamatic upper layer APIs and abstracts the actual management task from the underlying hardware devices.

In less complicated terms; it means that a program can say  “Hey, IMC change the VLAN on this port” and IMC, assuming IMC actually supports that particular device, it will change the VLAN on that port.

INDEPENDANT OF THE ACTUAL VENDOR

Yup. That’s right.  IMC doesn’t care if that device is a HP 5500EI ( comware ), a HP 3800 ( procurve), or even a Cisco Catalyst 3560.  From the perspective of the developer on the other side. It’s as simple as “Hey, IMC change the VLAN on this port”.

So the actual challenge I was given was the following.

” A customer wants to take a bar-code reader, scan in the MAC address of a device, plug it into the network, push a button and then have IMC automatically put it in the right VLAN”.

Now first I had to break that down into the various components.

1) Bar-code reader scans the MAC address

2) Program has to capture the MAC address for use in the %mac-address% variable in the script.

3) Find the device in the network

Hmmm… this could be more difficult than I thought.

So, I need to mock this up, so I break out to a windows CMD prompt.

Ping a known address ( my Synology NAS — LOVE THIS PRODUCT ).  And then put do a arp -a  to get the following output

10.101.0.51           00-11-32-10-03-8b

Now if I was using the IMC web-interface, I would just use the   Resource-Terminal Access-Real-Time Location feature which will, you guessed it, locate a host in real-time using the mac-address or the IP address.

Hmmm… that kinda sucks for output for the script to leverage.  So I went and looked at the eAPI documentation and came out with this little baby

The eAPI call is the following  ( if you wanted to search for an IP address you would use type=2 instead of 1 )

( Don’t click this, it won’t take you anywhere)

http://10.101.0.201:8080/imcrs/res/access/realtimeLocate?type=1&value=00-11-32-10-03-8b

The return is the following

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

<list>

<realtimeLocation>

<locateIp>00:11:32:10:03:8b</locateIp>

<deviceIp>10.101.0.221</deviceIp>

<ifDesc>GigabitEthernet1/0/21</ifDesc>

</realtimeLocation>

</list>

Yup. Good old XML. Easy to apply transforms or grab variables.  Programmers love this stuff.

4) So now I have the device IP (switch) that this is plugged into, and the ifDesc which is the actual interface it’s located on. So now I have to figure out how to apply the VLAN to this interface. So I break out the trusty eAPI documentation and start looking for the VLAN section.

Hmmm… I have the devIP and the ifDesc.. not the devID and the ifIndex

note to self: Feedback to the developers to have the first command return the devID and the ifIndex variables

So now I have to find the devID and the ifIndex for that devIP and intDesc

5a) Now if I was on the trusty IMC web interface, I would go to the device resource page… hmmm… that doesn’t appear to be there. I guess instead, let’s go to the eAPI documentation and look for something that looks like a dev query.

Yup. It’s actually called devquery. And it looks like I can filter based on the device IP.

Cool. So now I can search for the specific device IP and hope we get the devID variable back that we need for the VLAN call.

http://10.101.0.201:8080/imcrs/plat/res/device?ip=10.101.0.221

The return is the following

  <?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes” ?>

<list>

<device>

<id>15</id>     ———————————————- This is the device ID that we need to reference later

<label>HP_5500EI</label>

<ip>10.101.0.221</ip>

<mask>255.255.255.0</mask>

<status>1</status>

<statusDesc>Normal</statusDesc>

<sysName>HP_E5500EI</sysName>

<contact>HP Montreal</contact>

<location>Marlborough, MA 01752 USA</location>

<sysOid>1.3.6.1.4.1.43.1.16.4.3.36</sysOid>

<sysDescription>HP_5500EI</sysDescription>

<devCategoryImgSrc>Switch</devCategoryImgSrc>

<topoIconName>stack</topoIconName>

<categoryId>1</categoryId>

<symbolId>1022</symbolId>

<symbolName>HP_5500EI</symbolName>

<symbolType>3</symbolType>

<symbolDesc>HP_5500EI</symbolDesc>

<symbolLevel>3</symbolLevel>

<parentId>1003</parentId>

<typeName>3Com S4800G PWR 24-Port</typeName>

<mac>00:1e:c1:dc:fc:01</mac>

<link op=”GET” rel=”self” href=”http://10.101.0.201:8080/imcrs/plat/res/device/15” />

</device>

</list>

There it is.  DevID is  “15”.

5b) So now we need to figure out that ifIndex value associated with <ifDesc>GigabitEthernet1/0/21</ifDesc> that we pulled above.

If I was in the webinterface, I would simply go to the device ( 10.101.0.221 ), click on the interface list, click on interface Gig 1/0/21 and I would pull out the ifIndex from the interface…

But again, those programmers don’t want HTML, they want an easy XML output that they can play with. So let’s find that…

http://10.101.0.201:8080/imcrs/plat/res/device/15/interface?start=1&size=100

This returns a whole bunch of data for all the interfaces on the switch, but I’m sure that any decent programmer can write a regex expression to only return the one who’s ifDesc value is for Gig 1/0/21, right?

<interface>

<ifIndex>21</ifIndex>      ——————————————–In this case, the ifindex value is the same as the port number. That’s not always going to be true. This is the other variable for the set VLAN

<ifType>6</ifType>

<ifTypeDesc>ETHERNETCSMACD</ifTypeDesc>

<ifDescription>GigabitEthernet1/0/21</ifDescription>

<adminStatus>1</adminStatus>

<adminStatusDesc>Up</adminStatusDesc>

<showStatus>2</showStatus>

<statusDesc>Down</statusDesc>

<operationStatus>2</operationStatus>

<operationStatusDesc>Down</operationStatusDesc>

<ifspeed>10000000</ifspeed>

<appointedSpeed>-1</appointedSpeed>

<ifAlias>GigabitEthernet1/0/15 Interface</ifAlias>

<phyAddress>00:1e:c1:dc:fc:4f</phyAddress>

<mtu>1522</mtu>

<lastChange>4 day(s) 21 hour(s) 39 minute(s) 50 second(s) 990 millisecond(s)</lastChange>

<lastChangeTime>42359099</lastChangeTime>

<filterTrapStatus>0</filterTrapStatus>

</interface>

So now we have the DevID     15     and the ifIndex    for the actual interface where that MAC-address is located.

So let’s go back to the set that VLAN

Let’s assume that you wanted to put the device in VLAN 20,   you would run the following

http://10.101.0.201:8080/imcrs/vlan/20?devId=15&ifIndex21

That’s about it for the task. Now any decent programmer is going to have to put in some checking and error handling, for instance, you might want to check whether or not that VLAN actually EXISTS on the switch. ( Can’t put a VLAN on a port if the VLAN doesn’t exist on the switch, right? ) or maybe return an error if the MAC-address is actually seen on two interfaces, but in a nutshell that’s it.

note: I would also suggest that the dev actually bounce the port to make sure that the device hasnt’ gotten locked in with a DHCP address on the wrong subnet.

/plat/res/device/{deviceId}/interface/{ifIndex}/down   to down the interface

so for use that would be  /plat/res/device/15/interface/21/down

and then immediately do a

/plat/res/device/{deviceId}/interface/{ifIndex}/up

or again for us  /plat/res/device/15/21/up

Now whether the actual switch commands are “switchport acces vlan 20 ”  or ” port access vlan 20 ” or some other variation on a theme doesn’t actually matter to your devOps team. They just write the code to follow the steps and IMC and the eAPI will take care of the rest.

Pretty cool stuff. 🙂

@netmanchris