I sat down with my good friend Ethan Banks over at Packet Pushers for my second Heavy Networking podcast in hopes of discussing my JSON-MANIA!
Heavy Networking 563: Automating Documentation With Ansible, Genie, And Jinja2 – Packet Pushers
The modern approach to enterprise network management
I sat down with my good friend Ethan Banks over at Packet Pushers for my second Heavy Networking podcast in hopes of discussing my JSON-MANIA!
Heavy Networking 563: Automating Documentation With Ansible, Genie, And Jinja2 – Packet Pushers
In my opinion Network Access Control (NAC) using a mix of 802.1x and MAC Address Bypass (MAB) with dynamic Access Control Lists (dACL) is not just an extremely important foundation of a modern, high secure, enterprise network. It is actually a Software Defined Network (SDN) that completely changes the operationalization and dynamically adjusts the very configuration of your devices.
Cisco Identity Services Engine (ISE) uses Policy Sets to achieve this providing policy for Authentication – validating the identity a device claims to be – and Authorization – what, if anything, that identified device is permitted access to. Using either certificates and the 802.1x protocol or, if certificates are unfeasible, a MAC-based permission that pushes a dynamic ACL to the switch limited the access that MAC address has on the network.
I’ve implemented this at scale – and now I am trying to use automation tools to help operate, monitor, troubleshoot, and generally understand the impact of my ISE policy sets at the Access layer. And while ISE provides some amazing GUI-based tools – the Live Logs, Live Session, Policy Sets, and Context Visibility – if I can avoid using a GUI, sorting, filtering clicking, menu-driven system – I would like to just get right to the data!
In your ISE deployment options you can deploy Monitoring nodes / personas
These personas come with REST APIs !
But these are not to be confused with the ISE External REST Service (ERS) APIs!
(You can easily tell the difference in the API URL noting the presence of /mnt or the /esr path respectively)
In an Ansible Playbook
A lot of different automation and infrastructure as code tools were used to create this solution including:
Yes. For some reason ISE MnT Nodes do not return structured JSON at all and you can only receive XML format.
Unacceptable indeed!
Now that we’ve identified this limitation we have another step to consider in our automation orchestration – that is to parse and transform – the XML to JSON so we can work with it in Jinja2
ISE MnT API requires some specific permissions under a user account before you get going:
As with any API development I like to start with Postman.
Setup a Postman Collection called Cisco ISE
Add the username and password under Authentication – Basic Auth
Your GET string to get to the Active Sessions API is as follows:
https://ise.domain.com/admin/API/mnt/Session/ActiveCount
With very simple headers:
Resulting in the following data set:
Now we have the components we need – the working credentials, URL string, headers, and expected output – we can migrate the code over to Ansible using the URI module.
First we need to set up some variables we can use to connect to the ISE MnT API. I use a file called Enterprise.yml for this in group_vars
Then I can proceed with my playbook called CiscoISEActiveSessionTotals.yml
The first task is to use the URI module and register the results from the ActiveCount MnT API into a variable ActiveCount_XML
Next, because it is only a single value, we can simply use XML to parse out that field as follows and register the new variable FilteredCount
I like to always capture a .json file of the data as a RAW artifact of the unmanipulated data:
Which looks just like the Postman output:
It should be noted that the above file is the contents of FilteredCount exactly.
Now I can pass this along to the Jinja2 template:
Which looks like this:
And results in this:
Very nice!
Using this as a foundation can we transform the more advanced APIs that return more than a single XML value?
Let’s find out!
Adding the next API as a Request to the Postman Cisco ISE collection
Which returns a data set like this:
We will need another way, beyond XML and xpath, to parse this output. Ideally we could find a pre-made read-to-go XML to JSON conversion utility.
I tried, and failed, several different parsing tools including the Ansible recommended XML filter parse_xml with a spec file – but I just couldn’t figure this out.
Fortunately Ganesh Nalawade, Principal Engineer at Ansible, reached out to me with a great utility
So first we go out to the ActiveList REST API and register the ActiveList_XML
Now, after installing the Utilities from Ansible Galaxy, we simply feed the text ActiveList_XML.content – into the XML parser – and register the parsed data into a new variable ParsedActiveList
So lets copy that over to a .json file and take a look
Which results in:
Alright we have the XML parsed over to JSON! Now we can template it!
Looking at the JSON above we now need to loop over the activeSession under ParsedActiveList.parsed.activeList
In Jinja2 it looks like this:
Where we pick and choose our fields and the order we want to comma separate them. I like to include the | default(“N/A”) filter to add a value of “N/A” to any empty or non-present value.
The resulting CSV looks like this!
Now this is filtered against my user name but I have all 8,000+ authentication sessions, one per row, in this CSV file!
Easy filtered (as seen above) and sorted. Searchable. Simple.
What else can we do with this API?
Now in my pièce de résistance we are going to add another parsing technology I love – the Cisco Genie Parser – to capture MAC addresses to feed the last Cisco ISE MnT REST API – the MAC Session.
Add this final Request to your Cisco ISE Postman Collection replacing the MAC with either a Postman variable or a MAC that has a session in ISE.
Which returns:
Ok so now that we know we can send any MAC with a session in ISE against the REST API – how can we dynamically find and feed those MAC addresses from the network?
Answer: Genie
In order to get a list of authentication sessions on a Cisco switch we use the show authentication sessions command at the CLI
So the first thing I do is check the Genie Parser library and see if they have an available parser for the command and for what Cisco platforms they support.
Sure enough – there is a parser for the command I need.
So back to the Ansible playbook – first we need to run the ios_command and run the show authentication sessions command registering the results into a variable. This playbook needs to be scoped for the Access layer hosts where the 802.1x / MAB boundary is enforced.
Then we use the Genie Parser to transform the RAW CLI standard output (stdout) to – you guessed it – JSON ! We only set this fact (the variable) when the output does NOT equal “No sessions currently exist” in case there are no authentication sessions present on the device.
Now we don’t need this output in a JSON file but we do need to parse it, or query it, in order to get the MAC addresses from the results in order to send them to the ISE MnT API.
JSON_Query is another extremely potent Ansible filter that works like an SQL query but against the structured JSON. It can take some getting used to (painful laughing) but once you get the hang of the syntax it’s incredible fast and powerful.
I owe a big shout out to my pair-programming partner who eventually figured this out with (for?) me
To break this apart:
We set a variable up jquery to hold the actual query itself.
Then we set a fact which is the pyats_auth.interfaces value, which we:
Now we have another list, MACList, that contains a list of MAC addresses from the JSON_Query, which is querying the JSON we used Genie to convert from the IOS command!
From here it’s a simple matter of feeding the API each MAC in that list in a loop and registering the results:
I want to draw your attention to additional filters I had to use. Primarily because I was stuck on “Why isn’t this working? It should work .. but it doesn’t work” for a long time at this step. Eventually using debug I printed myself the variable MACList when I spotted something – the format of the MAC address!
The Cisco IOS MAC format is different than the Cisco ISE MnT REST API expects!
Meaning I was feeding the API:
ab00.cd11.ef22
Instead of:
AB:00:CD:11:EF:22
So by adding the hwadd(‘linux’) filter it changed the structure of the MAC address.
Then by adding the upper filter it changed the lower case letters to upper case.
Once I figured this out – it all started to work.
So now we have the XML in MACSessionDetails which we again need to parse with the Ansible Utility CLI_parse
Now we need another loop here to loop over each result and we can use the .content key being the {{ item }}
So now we can actually create our.json file from the parsed XML
Which looks like this:
Now Jinaj2 templating JSON lists are a funny thing. Because we get a natural list back, as indicated by the square bracket after results [ we can loop into this “directly”.
Meaning
{% for result in ParsedActiveSessionMACList.results %}
{{ result.parsed.sessionParameters.user_name }}
{% endfor %}
NOT
{% for result in ParsedActiveSessionMACList.results %}
{{ ParsedActiveSessionMACList.results[result].parsed.sessionParameters.user_name }}
{% endfor %}
Which returns:
Success!
What I find neat:
Part two of my podcast with Zig is out today and is all about Automation!
And make sure you enter Zig’s February draw for the best give away on the Internet!
Thanks Zig!
For whatever reason it seems to be difficult to get even just a “Star” review on Amazon – even harder to get actual comments – but today I received two more 4-5 star reviews!
Thanks again for the support! If you enjoyed the book – or if you didn’t enjoy the book – I would love it if you could take the time to add a star (ok 5 star) review or your feedback on Amazon !
Woke up to see this amazing post out in the wild today
I recently had the chance to be interviewed, via Twitter, by Charles Uneze !
I am really inspired by Charles’ own journey into IT and his passionate pursuit of his CCNA ! It reminds me of trying to get my CompTIA A+ / N+ all those years ago starting my own journey!
Anyway I really had a great time and Charles asked some great thought provoking questions !
Check it out and make sure to follow Charles’ journey !
My good buddy Zig – who hosted me on Zigbits podcast for a 2-part Network Design (Part One) and Network Automation (Part Two – coming in February) discussion.
As part of Part Two’s discussion – Zig’s first real conversation about Network Automation and Infrastructure as Code – Zig is having the most amazing giveaway I’ve ever seen!
THERE ARE 11 WAYS – INCLUDING A DAILY BONUS CLICK – TO ENTER TO WIN!
Good luck! This will change the winner’s career and life!
Cisco DevNet Code Exchange has published my repository !
Cisco_Facts (this link opens in a new window) by automateyournetwork (this link opens in a new window)
Ansible playbooks that use the IOS / NXOS Facts modules and Genie parsed commands to transform RAW JSON into business-ready documentation
Here you can easily start capturing Ansible Facts for IOS and NXOS and transform the JSON into CSV and Markdown !
Also included are a bunch of valuable Genie parsed show commands which transform the response into JSON then again transforms the JSON into CSV and Markdown!
The playbooks use Prompts so you should be able to clone the repo and update the hosts file and start targeting your hosts! For full enterprise support I suggest you refactor the group_vars and remove the prompts moving to full Ansible Vault – but for portability and ease of start-up I’ve made them prompted playbooks for now.
I would love to hear how they work out for you – please comment below if you have success!
I am very pleased to release the Cisco Services API Ansible Playbooks Version 2.0 which has been approved and released on Cisco DevNet Code Exchange !
You can find the code here and here
This major revision basically shifts away from lineinfile to Jinaj2 Templates for scale, performance, readability, and general best practices.
The Cisco Serial 2 Info API receives a valid serial number and then returns structured JSON with your Cisco Contractual information !
The playbook uses the Genie parser to parse the show inventory command
After authenticating against the OAuth 2 service to get a Bearer token
It provides the API the serial number for every part per device.
The API provides the following information back:
Which we first dump into JSON and YAML files
Then template into CSV and MD
Using Jinja2
Which gives us:
The other, very similar, Ansible playbook uses the Cisco Recommended Release API to create a spreadsheet with the current image on a host and the Cisco recommended version for that host given the Part ID (PID)
Here we don’t even have to use Genie to parse we can use the Ansible Facts module
And we transform again with Jinja2
And get this create report!
Please reach out to me directly if you need any help implementing these playbooks but I believe the instructions and code to be easy enough any beginner, with a little bit of refactoring and thought, could use this code as a starting point in their automation journey.
Cisco_API_v2 (this link opens in a new window) by automateyournetwork (this link opens in a new window)
Ansible playbooks that capture serial number and PID and send them to the Cisco.com APIs transforming the response into business-ready documents. Version 2.0 uses Jinja2 templates.