Tag: networkautomation

Posted on

Collaboration is the key to automation success – A Genie Success Story

I’ve been on about a three year journey with network automation and while I have had great personal and technical success – my organization and most of those outside my immediate day-to-day circle are still shackled to the ‘old’ way of doing things (primarily the CLI).

This year I decided to start a new program – a training session for those outside of my small development team – primarily targeted at the “operations” staff who can benefit the most from automation and infrastructure as code. This includes network operators, monitoring / NOC team members, IT Security staff, other developers, compute (server / storage) teams; everybody is welcome. We divided the calendar up into different teams with different recurring timeslots.

In advance I had written and tested a bunch of Code for the Campus Core – Ansible Facts and Genie parsed show commands – transforming the output into business-ready documentation. My plan was simple enough:

1. Ensure everybody was on the same page and had the same toolkit
* VS Code
* Git
* Azure DevOps repository bookmarked
* Various VS Code extensions
* A basic overview of main vs working branches in Git
* A basic outline on Ansible, YAML, Jinja2, and JSON

2. An operator would create a working development branch – in our case the Distribution Layer – so dist_facts Git branch.
* This operator ‘drives’ the whole session sharing their screen
* Step by step, line by line, refactoring (fancy way of saying copy-pasting) working code from the Core and updating it for the Distribution Layer as necessary
* Git clone, add, commit, push, and pull performed in both VS Code and Linux CLI
* Ansible playbook executed with a –limit against one building, then at scale after validating output
* Thorough tour of the JSON, YAML, CSV, MD, and HTML files after each run

3. Work through Ansible Facts and various Genie parsed commands to build up a source of truth

So far it has been a very successful approach with the two teams adopting Marvel superhero teams (TEAM: CAPTAIN AMERICA and TEAM: IRON MAN respectively) allowing me to create memes like this:

Image

Anyway – back to the point – today we had the following exchange:

Me: “So – we just parsed the show etherchannel summary CLI command and transformed the output into CSV files – amazing right! Any questions?”

Operator catching on quickly: “We use the show interfaces trunk command often to track down what VLANs are being carried on which interfaces – can we transform that into a CSV?”

Me, excited and proud of the Operator: “Amazing question and I’m glad you brought up a practical example of a command you use in the field all the time we can maybe transform into something a little more useful than CLI output!

Launch the Genie parser search engine (under available parsers on the left menu) and let’s see if there is a parser available

Bingo! Let’s do it!”

The playbook

In this example we are targeting the Campus Core.

The playbook is simple enough

Use the Ansible ios_command module to issue and register the show interfaces trunk command

Next, using the Genie filter plugin

Filter the raw variable and register a new variable with the Genie parsed results

Note you need to pass the parse_genie filter two arguments the command itself and the appropriate Cisco operating system

Next I like to create a Nice JSON and Nice YAML file with the parsed results as follows:

Which look like this:

And this:

I then use a loop to create a CSV and MD file from a Jinja2 template

The Jinja2 Template looks like this

Couple things to hightlight:

* We challenge the current iteration of the Ansible loop and when it is on “csv” we template a CSV file format otherwise (else) it will be “md” and we template a markdown file
* The For Loop is over each interface in the results
* Be careful with the CSV file you need to regex_replace the comma out of results because they are comma-separated which will throw off your CSV file. The markdown does not require any regex.

Which results in this amazing sortable, searchable, filterable, version controller, source controller, truthful, fact-based CSV file:

Now this example is just the singular logical Core but we will quickly refactor the code next week in our next session together and the operator who wanted this playbook will get a chance to write it ! Then we will have the interface trunk information for the entire campus automatically in spreadsheets!

The moral of this story is to collaborate. Ask your front-line operators how automation can help them. Do they have any frequently used or highly valuable CLI Commands they want transformed into CSV format? Would Ansible facts help them? And then show them how to do it so they can start writing these playbooks for themselves.

Posted on

It’s always DNS – but it doesn’t have to be!

DDI Automation – A collaborative live demonstration with BlueCat

I had the pleasure of collaborating with my friends Dana Iskoldski and Chris Meyer over at BlueCat exploring and automating DNS, DHCP, and IP Address Management (IPAM) (DDI) with the BlueCat Address Manager (BAM) and the BlueCat Gateway.

Infrastructure as Code and network automation is not always about switches and routers – there is incredible value to be found automating all the way up the stack into the critical Layer 7 services like DNS and DHCP; or transforming from Microsoft Excel-based IP address management to a fully automated solution.

In this 2-part series watch us explore and ultimately automate DNS using the BlueCat BAM and Gateway APIs with Postman, Ansible, and Python

Part One – BAM API Automation with Postman and Ansible

Part Two – BlueCat Gateway Automation

GitHub repository

An Ansible playbook that uses the BlueCat Address Manager API and creates human readable reports about all networks from the RESTful response
https://github.com/automateyournetwork/BlueCatAddressManagerNetworkFacts
2 forks.
3 stars.
0 open issues.

Recent commits:

Posted on

Prevent an Intrusion – Run the Recommended Version!

In the wake of some very high profile IT security breaches and state sponsored attacks using compromised software today I wrote some infrastructure as code Ansible playbooks to create some business-ready documentation to help us understand our Cisco software version footprint against what release the vendor recommends. It is very important to run “Safe Harbor” code in the form of the Gold Star release. These releases are as close as it gets to being bug-free, secure, tested, and supported in production environments.

The ‘old-way’ involved getting the Cisco Part ID (PID) or several PIDs and looking up the recommended release on Cisco.com using an ever deepening hierarchy of platforms, operating systems, and PIDs. At scale this is like a day’s worth of work to go gather all of this information and present it in a way the business can understand.

Building on my recent success with the Serial2Info Cisco.com API as well as Ansible Facts I thought this might be another nice use-case for business-centric, non-technical (not routes, IP addresses, mac addresses, etc), extremely important and critical insight.

Use Case

Can I automatically get the PID from a host or group of hosts and provide it to the Cisco.com Software Suggestion API building business-ready reports in CSV and markdown?

Answer: Yes!

The Playbook

Again you are going to need:

* A Linux Host with SSH access to your Cisco IOS devices and HTTPS access to the Cisco.com API
* Credentials for the host and for the OAuth2 API
* We are not using Genie parsers here so just “base” Ansible will work

Step 1. Setup credential handling

Create a playbook file called CiscoCoreRecommendedReleaseFacts.yml

Again I use prompted methodology here same as the Serial2Info API

Gather the username, enable secret, Cisco.com API ClientID, Client Secret

Step 2. Gather Ansible Facts

Using the ios_facts module gather just the hardware subset

Because we are using Ansible Facts we do not need to register anything – the JSON is stored in the Ansible magic variable ansible_facts

I need 2 keys from this JSON – the PID and ideally the current running version. These can be found as follows in the ansible_facts variable:

Which is accessed as ansible_facts.net_model

Which again is accessed as ansible_facts.net_version

With the information above – without going any further – I could already build a nice report about what platforms and running versions there are!

But let’s go a step further and find out what Cisco recommends I should be running!

Step 2. Get your OAuth2 token

First, using the Ansible URI module

We need to get our token using the registered prompted credentials.

The API requires the following headers and body formatting; register the response as a variable (token):

We have to break apart the RAW JSON token to pass it to the ultimate Recommended Release API:

Now we are ready to send PIDs to the API.

Step 3 – Send PID to Cisco.com API

Again using the URI module:

Here we pass the ansible_facts.net_model Fact to the API as an HTTP GET:

The headers and body requirements. Notice the authentication and how we pass the Bearer Token along. We also register the returned JSON:

Here is what the returned JSON looks like:

The highest level key is json or accessed via RecommendedRelease.json

There is a productlist

Which as you can see is a list as denoted by the [ ]

Inside this list is another product key with the values from the API about the product itself

A little further down we find the recommended software release

Step 4 – Transform technical documentation into business ready CSV / MD files

These JSON and YAML (I also use the | to_nice_yaml filter to create a YAML file along with the JSON file) files are create for technical purposes but we can do a bit better making the information more palatable using business formats like CSV and mark down.

It is just a matter of using Jinja2 to template the CSV and Markdown files from the structured JSON variables / key-value pairs.

Add a final task in the Ansible playbook that will loop over the CSV and MD file types using the template module to source a new .j2 file – CiscoCoreRecommendedReleaseFacts.j2 – where our logic will go to generate our artifacts.

The Jinja2 starts with an If Else EndIf statement that checks if the Ansible loop is on CSV or not. If it is it uses the CSV section of templated file format otherwise it uses markdown syntax.

First we want to add a CSV header row

Then we need a For Loop to loop over each product in the productList

Now we add our “data line” per product in the loop using the various keys

Hostname for example uses the Ansible magic variable inventory_hostname

Then we want the Base PID. We use the Ansible default filter to set a default value in case the variable happens to be empty.

We continue accessing our keys and then we close the loop.

Now we need to create the Markdown syntax

And the same logic for the “data row” but with pipes instead of commas. Make sure to close off the If statement

Step 5 – Run playbook and check results

We run the playbook as ansible-playbook CiscoCoreRecommendedReleaseFacts.yml

Answer the prompts

Let the playbook run and check the results!

Summary

Again with a few free, simple tools like Ansible and the Cisco.com API we can, at scale gather and report on the current running version and the vendor recommended version quickly and easily and fully automatically!

Now go and start protecting your enterprise network armed with these facts!

Posted on

Sign on the Dotted Line

Layer 9 issues – finance – are often some of the most challenging a network engineer faces. Contract management can be particularly difficult in any scale organization especially if you are not “sole source” purchasing. Serial numbers and contracts are also not typically things the “network people” want to deal with but when that P1 hits and you try to open a SEV 1 TAC CASE – only to find out you are not under contract – I’ve been in less terrifying car accidents than this nightmare scenario.

I have good news ! Using a mix of automation and developer-like tools the network engineer can now create a real source of truth that, along with routes and MAC address-tables and other technical information, can include inventory and contractual business documentation from stateful, truthful, real-time, facts from Cisco.

Ok so let’s get into it!

As a rough outline for our logic here is the use case:

Can I automatically gather the serial numbers from Cisco device hostnames and then provide them to Cisco and get my contractual state for each part on each device?

Answer: YES !

What you will need:

* Linux host with Ansible, Genie parser
* Linux host requires both SSH access to the Cisco host and Internet Access to the OAuth2 and Cisco.com API HTTPS URLs
* Cisco SmartNet Total Care – I have written up instructions in this repo under the “OnBoarding Process” section

The Playbook

Step 1 – We will need to get the serial number for every part for a given hostname. For this we will use the standard show inventory command for IOS using the Ansible ios_command module. I will be using prompted methods for demonstration purposes or for on-demand multi-user (each with their own accounts) runtime, but we could easily Ansible Vault these credentials for fully hands-free run time or to containerize this playbook. I am also targeting a specific host – the Core – but I could easily change this to be every IOS device in the enterprise. This playbook is called CiscoCoreSerial2InfoFacts.yml

First prompt for username, enable secret, Cisco Customer ID, Cisco Customer Secret and register these variables:

Then run the ios_command show inventory and register the results in a variable.

Step 2 – Parse the raw output from the IOS command

Next, we use Genie to parse the raw results and register a new variable with the structured JSON. Genie requires, for show inventory, the command, the operating system, and the platform (in this case a Cisco 6500)

And here is what that structured JSON looks like:

So now we have a nice list of each part and their serial number we can feed the Cisco.com API to get back our contract information.

Step 3 – Get an OAuth 2 token from Cisco web services.

Cisco.com APIs use OAuth2 for authentication meaning you can not go directly against the API with a username and password. First you must retrieve a Bearer Token and then use that limited time token within it’s lifetime against the ultimate API.

Using the Ansible URI module go get a token and register the results as a variable. Provide the Customer ID and Client secret prompts to the API for authentication. This is an HTTP POST method.

With the new raw token setup the token type and access token from the raw response

Step 4 – Provide token to the Serial2Contract Cisco API to get back contractual information for each serial number.

Now that we have a valid token we can authenticate and authorize against the Cisco SmartNet Total Care Serial Number 2 Contract Information API.

In this step we are going to use an Ansible loop to loop over the Genie parsed structured JSON from the show inventory command providing the sn key for each item in the list. We need to use the Python | dict2items Ansible filter to transform the dictionary into a list we can iterate over.

The loop is written as

loop: “{{ pyats_inventory.index | dict2items }}”

And each serial number is referenced in the URL each iteration through the loop:

url: https://api.cisco.com/sn2info/v2/coverage/summary/serial_numbers/{{ item.value.sn }}

We register the returned structured JSON from the API as Serial2Info which looks like this:

So now I have the JSON – let’s make it a business ready artifact – a CSV file / spreadsheet and a markdown file – using Jinja2

Step 5 – Using Jinja2 lets template the structured JSON into a CSV file for the business.

Create a matching Jinja2 template called CiscoCoreSerial2InfoFacts.j2 and add a task to Ansible that uses the template module to build both a CSV file and a markdown file from the JSON.

In the Jinja2 file we need a section for CSV (if item = “csv”) and a section for markdown (else) based on their respective syntax. Then we need to loop over each of the responses.

result in Serial2Info[‘results’] is the loop used. I also add a default value using a filter | default (‘N/A’) in case the value is not defined. SFPs for example do not have all of the fields that a supervisor module has so to be safe it’s best to build in a default value for each variable.

The final Jinja2 looks something like this:

Which results in a CSV and Markdown file with a row for every serial number and their contractual facts from the API.

Summary

Large scale inventory and contract information can easily be automated into CSV spreadsheets that the business can easily consume. Ansible, Genie, Cisco.com APIs, Jinja2 templating and a little bit a logic come together into an automation pipeline that ensures contractual compliance and inventory fidelity at scale!

Posted on

Ansible Performance – Moving to Jinja2 for Automated Documentation

When Ivan Pepelnjak has advice for you – take it!

I wrote a post about untangling dynamic nested loops in Ansible.

In another recent post about trying to improve Ansible performance I didn’t get very far – but this could be the silver bullet I’ve been looking for to both optimize and make my Fact / Genie parsing playbooks more elegant code but also to bring my run times down so I can bring this from the lab to production.

Jinja2 Templates

One of the reasons why I perked up at Ivan’s generous suggestion is because I am a big fan and heavy user of Jinja2 templates already to generate intended configurations (Cisco IOS, NXOS configurations; JSON files for API POST) and documentation (intended configs in CSV, markdown, and HTML) – but I had just never thought of implementing them to create my documentation from received data!

My old way involved taking the structured JSON and using lineinfile or copy to create my output files. This was slow. Very slow.

Copy method:

Line In File method:

How to refactor this?

So I already have everything I need content wise – a header row and the data rows – I just need to move this into Jinja2 format. As it turns out there are some added benefits beyond just performance that I will highlight.

My quick use case was my CiscoNXOSFacts.yml playbook against 2 7Ks just gathering facts (nxos_facts) and transforming the structured JSON into business documentation.

– Create Nice JSON file from facts – Ansible | to_nice_json filter
– Create Nice YAML file from facts – Ansible | to_nice_yaml filter
– Create CSV file from facts
– Create markdown file from facts
– Generate HTML from markdown

So the first refactoring is the actual task from using copy or lineinfile to using template. Template needs a source (a new Jinja2 template file we will create in our next step).

Template also needs a destination. Here is where we can use the programmatic capabilities of Jinaj2 to simplify, optimize, and massively improve performance by setting up a simple loop and create both files. Wait files plural? Yes. My old way involved creating 2 separate files in 2 separate tasks. Now that I am using Jinja I can use variables – one item being “csv” and the other item being “md” – and pass them to the template for processing.

So create a Jinja2 template file called CiscoNXOSFactsTemplate.j2 to create your CSV and Markdown files.

Before I show the template I want to highlight another massive improvement to using Jinja2 – Jinaj2 is able to iterate naturally over dictionaries while my previous method had to pass the structured JSON through the | dict2items Ansible filter (against adding processing time). This simplifies the code quite a bit.

In the template we will test if the loop is on csv or md and create either a csv or md formatted output file.

Else if item is md create the markdown file format

One last and very important comment and benefit of Jinja2 is that I do not need to use Regular Expressions “as much” to clean up the JSON. | dict2items leaves a lot of garbage JSON characters behind which I had to previously use processor intensive RegEx tasks to clean up. Now Jinja2 does this cleanup and conversion from RAW to Nice JSON for me!

Results

I have only tested 1 playbook but I am very excited about this new refactored code !

Again this playbook “only” touches 2 physical devices but I have playbooks that potentially could be gathering facts and generating artifacts for hundreds of devices. But the results are pretty clear particularly the system time

Old way:

New way:

So roughly half the “real” time but look at the system time – from 36 seconds down a third to 12 seconds! WOW!

Thanks again!

A big thanks to Ivan for taking the time to comment and point me in a better direction. You may not know this but when I started my automation journey one of my resources along with several books, Cisco DevNet, trial and error, was my IPSpace.net subscription. If you are looking for a very affordable and very comprehensive library of networking and automation knowledge this is a good place to start.

Posted on

Dynamic Loops in Ansible

I’ve done many great things with Ansible but occasionally I come across a logical problem that may stretch the tool past it’s limitations. If you have been following the site I am on a big facts discovery and automated documentation movement right now using Ansible Facts and Genie parsers.

The latest parser I am trying to convert to documentation is the show access-lists command.

So we use the Ansible ios_command module and issue the command then parse the response. This is a playbook called CiscoACLFacts.yml

I always start with creating Nice JSON and Nice YAML from the structured JSON returned by the parsed command:

Then I examine the Nice JSON to identify “how” I can parse the parsed data into business documentation in the form of CSV, markdown, and HTML files.

And I have my CLI command converted into amazing structured human or machine readable JSON:

So here is where the logical challenge comes into play with Ansible.

As you know an Access Control List (ACL) is a list (hint: it’s right in the name) of Access Control Entries (ACEs) which are also a list. Both the ACL and the ACE are variable – they could be almost anything – and I need 2 loops: an outer loop to iterate over the ACLs (shown above is the “1”) and an inner loop to iterate over the ACEs (shown above as the “10”).

So I brush up on my Ansible loops. Looks like I need with_nested.

Problem: Ansible does not support dynamic loops as you would expect. Here is what I “wanted to do” and tried for a while before I figured out it wasn’t supported:

with_nested:
– “{{ pyats_all_access_lists | dict2items }}”
– “{{ item[0].value.aces }}”

No go. “Item not found” errors.

Here is the work around / solution:

Before I get into the loops a couple things to point out to anyone new to infrastructure to code or JSON specifically. The Genie parsed return data is not a list by default. Meaning it cannot be iterated over with a loop. We have to filter this from a dictionary – as indicated in JSON by the { } delimiters – into a list (which would be indicated by [ ] delimiters in the JSON if it was a list we could iterate over) – before we can loop over it.

| dict2items is this filter.

The loops:

You can define the outer loop key using loop_var as part of loop_control along with include to build a dynamic outer / inner loop connection.

In order to create my CSV file:

1 – Delete the file outside the loops / last Ansible task before entering the loops

2 – * Important new step here *

We need to perform the outer loop, register the key for this outloop, and then include a separate YAML file that includes the inner loop task

3 – * Another important new step here *

Create the referenced file CiscoACLInnerLoop.yml with the inner loop task, in this case, the task to add the rows of data to the CSV file

Things to identify in the above task:

The loop – it is using the outer loop (the loop_var) ACL_list as the primary key then we turn the .value.aces dictionary into another list with | dict2items giving us the inner list we can iterate over.

Important – the inner loop is what Ansible will reference from this point forward meaning item. now references the inner items. In order to reference the outer key you need to reference the loop_var again as seen on the line: “{{ ACL_list.key }},{{ item.key }}

This gives us the ACL then the individual ACE per row in the CSV file! Mixing the outer and inner loops!

Recommendation – you will notice the start of an {% if %} {% else %} {% endif %} statement – because almost everything in an ACL and ACE list is variable you should test if each item.value.X is defined first, use the value if its defined, otherwise use a hard coded value. As such:

{% if item.value.logging is defined %}

{{ item.value.logging }}

{% else %}

No Logging

{% end if %}

Next, back in the main playbook file, outside the loop, we finally add our CSV header row:

For the sake of keeping this short there is likely some Regular Expression replacements we need to make to clean up any stray JSON or to remove unnecessary characters / strings left behind but in essence we have the follow show access-lists command rendered into CSV:

Operations, management, compliance and standard, and most of all IT SECURITY is going to love this! All of this is in a central Git repository so all of these artifacts are Git-tracked / enabled. All of the CSV files are searchable, sortable, filterable, and EXCEL ready for the business!

Summary

Before you give up on any problem make sure you find and read the documentation!

I have to revisit some previous use cases and problems now with fresh eyes and new capabilities because I had given up on transforming some previous dictionaries in dictionaries because I didn’t know what I was doing!

One step closer! I hope this article helped show you how dynamic Ansible looping is done and you don’t have to fail and struggle with the concept like I did. I am out there on Twitter if you have any questions!

Posted on

Can we make Ansible go faster?

When I describe Ansible to people I tend to use many positive adjectives. Amazing, incredible, easy, revolutionary, powerful, and a few others. One adjective I never use, however, is fast. I would not describe Ansible as a high performance tool. Compared to manually doing the things I’ve come to automate with Ansible there is still no doubt I am saving hours if not days of effort. But now that I’m using Ansible for almost everything and at scale it would be great if I could get better performance out of the tool.

Over the years I’ve learned to run the ansible-playbook command then – and chant it like the late night informercial – “Set it and forget it!”

Its the one, sometimes painful, drawback I can find with Ansible. There has yet to be an infrastructure problem I have not been able to solve with Ansible – provided I am comfortable with waiting. “How long does this take?” change managers or operations will ask. “A while.” Is usually as optimistic as I can be.

(Note: It is a bit ironic sometimes the same crowd with complaints about how long a playbook takes to run are usually the same people who were comfortable with pre-automation manual-at-the-cli-of-every-device execution times into the days or weeks. Now anything more than a 10 minute playbook run seems like a long time. Go figure)

TL:DR

– Ansible is an amazing automation tool
– Ansible is not known for its performance
– Three modifications tried to make it go faster
– LibSSH
– Forks
– Pipelines
– No real improvements found with any of the above

Moving to LibSSH

The driving factor for me to bring my Ansible ecosystem into the shop and put it up on the lift to get underneath and into the mechanics of the configurations is this latest official blog post from Ansible.

“Not only is the new LibSSH connection plugin enabling FIPS readiness, but it was also designed to be more performant than the existing Paramiko SSH subsystem.”

This particular section of the blog post is what drives my exploration today. And yes FIPS readiness is important to me – the hook for me here is “designed to be more performant” – and yes the link they provide is great but I want to take the Pepsi Challenge myself.

Playbooks tested

I will be using the following playbooks with 2 different scale sets.

Cisco IOS Facts – Against my Lab distribution layer (4, Cisco 4500s) and my access layer (about 20 – 25 devices of various Catalyst flavours (2960, 3560, 3750, 3850, 9300)).

Cisco NXOS Facts – Same idea but against NXOS. 2 Nexus 7000 and 2 Nexus 5000.

The above playbooks use the Ansible facts modules. Let’s do some Genie Parsing of a show command as well.

IOS Genie show ip interface status

Methodology

In Linux you can use the time keyword command and prepend any command. Linux then provides three different timer – the real time, the user time, and the system time – results showing how long the command took to execute.

Result Set #1 – Defaults

With no changes to default Ansible here are the results. I will be standardizing on the sys results because of the input and other factors the real times and users times may have deviations:

Install LibSSH and modify Ansible

First step is to pip install the Ansible library we need:

Then we to update our persistent connections:

Refresh your Git repo and re-run the playbooks.

Result Set #2 – LibSSH

This image has an empty alt attribute; its file name is image-72.png
This image has an empty alt attribute; its file name is image-74.png
This image has an empty alt attribute; its file name is image-75.png
This image has an empty alt attribute; its file name is image-77.png
This image has an empty alt attribute; its file name is image-81.png

Forks

Ansible can be set to fork which allows multiple independent remote connections simultaneously.

Forks are intelligent and will only fork for the maximum number of remote targets. So I will set my forks in ansible.cfg to 50.

Now I don’t think this will help playbooks with under 5 targets because I believe Ansible defaults to 5 forks but maybe this will improve the Access Layer Facts which targets around 25 hosts. So lets just test against that one playbook.

Results Set #3 – Forks

This image has an empty alt attribute; its file name is image-74.png

Pipelining

Enabling pipelining reduces the number of SSH operations required to execute a module on the remote server, by executing many ansible modules without actual file transfer. According to the documentation this can result in a very significant performance improvement when enabled.

Add pipelining=true to the SSH connection section in ansible.cfg:

Result Set #4 – Pipelining

This image has an empty alt attribute; its file name is image-72.png
This image has an empty alt attribute; its file name is image-74.png
This image has an empty alt attribute; its file name is image-75.png
This image has an empty alt attribute; its file name is image-77.png
This image has an empty alt attribute; its file name is image-81.png

Summary

I didn’t have much success making Ansible go any faster either with the new LibSSH library, with forking, or with pipelining. I absolutely love ansible but the above times are from my small scale lab – imagine these run times in production at 5-10x the scale.

Have you found a way to make Ansible go faster that I’ve overlooked? Drop my a line!

Sean also jumped in to mention the driver for LibSSH was FIPS not performance and there are some performance improvements coming soon! Great!

Posted on

Excitement

And he’s right! I can’t stop making new GitHub repositories with Genie parsed show commands to documentation! Like show ip interface brief as seen above!

An Ansible playbook that uses Genie parser to convert Cisco IOS show ip interface brief command to documentation
https://github.com/automateyournetwork/IOS_GenieParsed_Show_IP_Int_Brief
0 forks.
2 stars.
0 open issues.

Recent commits:

Posted on

Letting the Genie out of the bottle

Imagine if you could transform that unstructured Cisco show command screen output into something more useful than just CLI output.

What if there was a way to transform an SSH CLI show command’s response into a RESTful API-like HTTP GET request response in RESTful API-like structured JSON?

Sounds amazing right? Well with Genie this is totally possible. I mentioned the CTAS Genie / pyATS / xPresso solution in My Toolkit post. I also suggested that gathering facts with Ansible is the best place to start with network automation and infrastructure as code.

But the Ansible facts, while impressive, rich, plentiful, and extremely useful, they do not necessarily contain all of the state information that IOS / NXOS CLI show commands provide. Some information, such as CDP neighbors, interfaces, IP addresses, is available with only the ios_facts / nxos_facts modules but for things like the configured Virtual Route Forwarders (vrf) on a router, the IP Address Resolution Protocol (ARP) tables, or the OSPF routing tables you are stuck with crappy old Cisco CLI output right?

Wrong. You now have a magical Genie on your side who is here to grant all your state capture and transformation wishes! And you get more than 3!

TL;DR

– The historic restrictions of using Cisco IOS / NXOS show commands as CLI-only, raw screen / putty logged output, have been lifted.
– Genie parsers provide REST API HTTP GET-like responses to common CLI show commands.
– Ansible integrated allowing for running and parsing show commands at scale.
– I like to create RAW JSON, Nice JSON, Nice YAML, CSV, Markdown, and interactive HTML mind maps from any JSON I can get my hands on. Now I can do it with Cisco show commands!
– Fill the gaps from what is missing from base Ansible facts.
– Build a powerful, dynamic, state aware documentation set for every device in your network from every day show commands.
– Not as difficult as you might think.
– Another modern network automation, infrastructure as code, tool network engineers should include in their skillset.
– The best development team in the business. The Genie / pyATS / xPresso developers have personally helped me out. Find them on Cisco WebEx Teams for often real-time responses.

What is Genie?

Genie is a parser that automatically converts Cisco IOS / NXOS command output into structured JSON. This structured JSON allows developers to then work more programmatically with the output from important, but otherwise useless, show command output.

For example I am using Genie to parse some key show commands and create a dynamic automated library of documentation in different formats.

You can also go a step further with pyATS and write boolean tests (true / false) in Python using the Genie parsed data as your source of test data. The show ip ospf neighbor command, for example, can be executed, parsed with Genie, and then tested with pyATS! All of this can then be wrapped in business logic, scheduling, and protected with RBAC in xPresso.

Amazing but I am not made of money – how much does all this capability cost?

It is all free.

How do I integrate it with Ansible?

The amazing Ansible integration that I am using is thanks to Clay Curtis and his incredible contributions. Thanks to Clay there are two installation steps on top of the standard Ansible installation and an open Python filter plugin – then you are ready to go.

Please visit the Ansible Galaxy role, Cisco DevNet Code Exchange, and GitHub repository for all the details.

Show and Tell

It’s easier to just demonstrate how the Parser can be used with Ansible. Some prerequisites:

– Linux host
– pip install ansible
– pip install genie
– ansible-galaxy install clay584.parse_genie
– SSH access to network devices from this host
– Credentials for the device (prompted)
– The parse_genie Python filter_plugin
– Make sure your ansible.cfg specifies the location of the parse_genie file in filter_plugins.

[defaults]
filter_plugins=../filter_plugins

Example: show vrf

Recall what a show vrf looks like at the CLI:

This could spawn for pages depending on how many VRFs are hosted on the router. Also – how do you work with this output? Log to a putty file and inspect in notepad offline? Not very user friendly.

Let’s Genie parse that same command and see what the output looks like as structured JSON and take the Pepsi Challenge against the CLI.

In a playbook called CiscoVRFFacts.yml I first scope the playbook (CampusDistribution), then prompt for username and password. Note the collection includes Clay’s clay584.genie collection.

Next I run my standard Cisco show command with the ios_command module and register the response (which is RAW unparsed IOS config at this point) Nothing fancy here.

The next step is where we use the filter_plug in to parse the registered raw response and register a new variable that holds the parsed output. Again – this is not very complicated once you understand the syntax.

Note the parsed command is the same as the original command, in this case show vrf, and we have to specify the operating system (Cisco IOS).

You can optionally print these parsed facts, the nice JSON, to the screen.

Resulting in something like this:

We can save this output to a variety of files and manipulate the output in a variety of ways previously unavailable to us with raw standard CLI IOS output.

For starters lets put the RAW JSON response into a JSON file with the Ansible copy module.

Which is good for forensics, audits, security purposes, or for downstream systems that intake raw JSON, but it’s not very human readable.

Add the Ansible filter | to_nice_json to get the output to look like the output that is printed to the screen.

Now this is up for debate but personally I prefer and find YAML even more human-readable than JSON. Let’s make a YAML file with the | to_nice_yaml filter.

As a reminder this is what the show vrf command looks like at the CLI:

This image has an empty alt attribute; its file name is image-53.png

Now, in YAML:

Incredible!

Going a step further we can try to manipulate the output for business suitable reports in CSV, markdown, and HTML files.

Using yet another Ansible filter, the dict2items, which as the name implies transforms a dictionary to a list of items, we can loop over the original variable {{ pyats_all_vrfs.vrfs }} key and create our CSV / markdown.

(There are some Regular Expression (RegEx) steps that clean up the JSON a bit omitted for brevity)

Add a header row.

And now you have a CSV file!

Similar steps can create a markdown.

And then an HTML mind map can be generated.

Look at all the business and operational value we’ve squeezed out of a simple show vrf command!

All of this code is available on Automate Your Network’s GitHub.

An Ansible playbook that uses the Genie parser to transform the Cisco IOS "show vrf" command into human readable documentation
https://github.com/automateyournetwork/IOS_GenieParsed_Show_VRF
0 forks.
1 stars.
0 open issues.

Recent commits:

Example: show ip arp

Start with the CLI show ip arp command output, which to be fair isn’t the worst CLI output around, which provides the ARP table for the Global Routing Table.

With more or less the same steps transform this into the same reports.

Setup the playbook:

Run the show ip arp command:

Parse it:

Create your 3 base RAW JSON / Nice JSON / Nice YAML files:

Check out this nice output!

Now anybody, CCNA level or not, can read the ordered structured list and see that VLAN20 has 1 neighbor with an IP of 172.24.2.1, the age, and the MAC address.

Similar steps to transform the output create the CSV / markdown / mind maps:

Also available on GitHub.

An Ansible playbook that uses the Genie parser to create reports from the show ip arp Cisco IOS command
https://github.com/automateyournetwork/IOS_GenieParsed_Show_IP_ARP
1 forks.
1 stars.
0 open issues.

Recent commits:

Example: show ip arp vrf {{ vrf }}

The exact same steps can be performs by simply adding show ip arp vrf <vrf name> with the same output as the Global Routing Table.

As a development aside I had big plans for show ip arp vrf {{ vrf }} to be a dynamic and automatically loop over all of the VRFs present on the route. I got pretty far but the parser itself hung me up.

Meaning I had a previous loop over the Genie parsed show vrf command which provided me the VRF name to feed the show ip arp vrf command. This all worked out and I could get the raw unparsed list like this:

ios_command:
commands:
– show ip arp vrf “{{ item.key }}”
loop: “{{ pyats_all_vrfs.vrf | dict2items }}”

But when it came time to parse this the following didn’t work.

| parse_genie(command=’show ip arp vrf {{ item.key }}’, os=’ios’)

I think because the parser is treating {{ item.key }} as raw text / the raw command and is not aware of the outer loop and to treat it like a variable. For the same reason I couldn’t write it to prompt for a VRF name. So, yes, I found one edge case drawback where I have to hardcode the VRF. Leave me a note if you see a way around this.

Summary

Genie parsers allow network engineers and operators to transform simple Cisco show commands into REST API-like requests with corresponding structured JSON response from the switch. This all magically happens behind the scenes allowing developers to actually make use of the show command output.

The days of setting up your Putty session to log your console to text files for offline analysis of raw standard output are finally over. Now you can work with JSON which in turn can be transformed into YAML, CSV, markdown, and HTML easily.

Along with Ansible facts, Genie parsed state data can solve the lifelong challenge of creating and maintaining good documentation.

Bonus code – show etherchannel summary

I couldn’t help myself and wrote another GitHub repository after finishing the blog. That’s how much I love Genie and how quick and easy it is!

An Ansible playbook that uses the Genie parser to transform Cisco IOS show etherchannel summary into usable reports.
https://github.com/automateyournetwork/IOS_GenieParsed_Show_Etherchannel_Summary
0 forks.
2 stars.
0 open issues.

Recent commits: