Is Your Network At Risk – Automating the Cisco PSIRT with Genie and Ansible

The security of my network keeps me up at night. Honestly it does. We live in a world where enterprise networks are defending themselves against state-sponsored attacks. But what do attackers look for? Typically, open, well-known, vulnerabilities.

Well, at least to the hackers, attackers, and even script-kiddies, these vulnerabilities are well-known. Those playing defense are often a step or two behind just identifying vulnerabilities – and often times at the mercy of patch-management cycles or operational constraints that prevent them of addressing (patching) these waiting-to-be-exploited holes in the network.

What can we do about it?

The first thing we can do is to stay informed ! But this alone can be a difficult task with a fleet of various platforms running various software versions at scale. How many flavours of Cisco IOS, IOS-XE, and NXOS platforms make up your enterprise? What version are they running? And most importantly is that version compromised?

The old way might be to get e-mail notifications (hurray more e-mail!), maybe RSS-feeds, or go device-by-device, webpage-by-webpage, looking up the version and if it’s open to attack or not.

Do you see why, now, how an enterprise becomes vulnerable? It’s tedious and time-intensive work. And the moment you are done – the data is stale – what, are you going to wake up every day and review threats like this manually? Assign staff to do this? Just accept the risk and do they best you can trying to patch quarterly ?

Enter: Automation

These types of tasks beg to be solved with automation !

So how can you do it?

Let’s just lay out a high level, human language, defined use-case / wish list.

  1. Can we, at scale, go get the current IOS / IOS-XE / NXOS software version from a device?
  2. Then can we send that particular version somewhere to find out if it has been compromised ?
  3. Can we generate a report from the data above?

Technically the above is all feasible; easy even!

  1. Yes, we can use Ansible and the Cisco Genie Parser to capture the current software version
  2. The Cisco Product Security Incident Response Team has incredible, secure, REST APIs available that we can automate with the Ansible URI module
  3. Using Jinja2 templates we can craft business-ready reports

Getting Started

First you need to setup your Cisco.com REST API suite access

Pre-Ansible Development

As with any new REST API I like to start with Postman and then transform working requests into Ansible playbooks.

First, under a new or existing Cisco.com Collection, add a new request called IOS Vulnerabilities

Cisco.com uses OAuth2 authentication mechanism where you first must authenticate against one REST API (https://cloudsso.cisco.com/as/token.oauth2) which provides back authorization Bearer token used to then authenticate and authorize against subsequent Cisco.com APIs

Your Client ID and Secret are found in the API portal after you register the OpenVuln API

Request and use a token from the API in Postman:

Now add your request for IOS

https://api.cisco.com/security/advisories/ios?version=

Test it out!

Let’s hard code a version and see what flaws it has

Ok so it has at least 1 open vulnerability!

Does it tell us what version fixes it?

Ok let’s check that version quickly while we are still in Postman

This version has no disclosed vulnerabilities!

One interesting thing of note – and our automation is going to need to handle it – is that if there are no flaws found we get a 404 back from the API not a 200 like our flawed response!

The Playbook

For the sake of the example I am using prompted inputs / response but these variables could easily be hardcoded and Ansible Vaulted.

So first prompt for your Cisco hosts username and password and your Cisco.com ClientID and Client Secret

Register the response

Then in the IOS.yml group_vars I have my Ansible network connections

I’ve put all IOS-platforms in this group in the hosts file to target them

Next step run the show version ios_command and register the reponse

Genie parse and register the JSON

Now we need, just like in Postman, to go get the OAuth2 token but instead of Postman we need to use the Ansible URI module

Then we have to parse this response and setup our Bearer and Token components from the response JSON.

Now that we have our token we can authenticate against the IOS open Vulernability API

There are a couple of things going on here:

  1. We are passing the Genie parsed .version.version key to the API for each IOS host in our list
  2. We are using the {{ token_type }} and {{ access_token }} to Authorize
  3. We have to expect two different status codes; 200 (flaws found on a host) and 404 (no flaws for the host software version)
  4. I’ve added until and delay to slow down / throttle the API requests as not to get a 406 back because I’ve overwhelmed the 10 requests per second upper limit
  5. We register the JSON response from the API

As always I like to create a “nice” (easy to read) version of the output in a .json file

Note we need to Loop over each host in our playbook (using the Ansible magic variable ansible_play_hosts) so the JSON file has a data set for each host in the playbook.

Lastly we run the template module to pass the data into Jinja2 where we will create a business-ready CSV file

Which looks like this broken apart:

Like building the JSON file first we need to loop over the hosts in the playbook.

Then we check if the errorCode is defined. You could also look at the 404 status code here. Either way if you get the error it means there are no vulnerabilities.

So add a row of data with the hostname, “N/A” for most fields, and the json.errorMessage from the API (or just hardcode “No flaws” or whatever you want here; “compliant”)

Now if the errorCode is not defined it means there are open flaws and we will get back other data we need to loop into.

I am choosing to do a nested set of two loops – one for each advisory in the list of advisories per software version. Then inside that loop another loop, adding a row of data to the spreadsheet, for each BugID found as well (which is also a list).

There are a few more lists like CVEs for example – you can either loop of these as well (but we start to get into too much repetition / rows of data) – or just use regex_replace(‘,’,’ ‘) to remove all commas inside a field. The result is the list spaced out inside the cell sorted alphabetically but if you do not do this it will throw off the number of cells in your CSV

The results

What can you do with “just” a CSV file?

With Excel or simply VS Code with the Excel Preview extension – some pretty awesome things!

I can, for example, pick the ID field and filter down a particular flaw in the list – which would then provide me all the hosts affected by that bug

Or pick a host out of the list and see what flaws it has, if any. Or any of the columns we have setup in the Jinja2

This image has an empty alt attribute; its file name is image-108.png

Included in the report is also the SEVERITY and BASE SCORE for quick decision making or the Detailed Publication URL to get a detailed report for closer analysis.

Which looks like this:

Automation is much more than configuration management. It can be used to manage, understand, and mitigate risk as well.

Together we can secure our enterprise networks and hopefully sleep a little more sound knowing they are safe.

Cisco Facts – A Collection of Ansible IOS / NXOS Facts and Genie Parsing Playbooks

Cisco DevNet Code Exchange has published my repository !

Dark Mode

Cisco_Facts (this link opens in a new window) by automateyournetwork (this link opens in a new window)

Ansible playbooks that use the IOS / NXOS Facts modules and Genie parsed commands to transform RAW JSON into business-ready documentation

Here you can easily start capturing Ansible Facts for IOS and NXOS and transform the JSON into CSV and Markdown !

Also included are a bunch of valuable Genie parsed show commands which transform the response into JSON then again transforms the JSON into CSV and Markdown!

The playbooks use Prompts so you should be able to clone the repo and update the hosts file and start targeting your hosts! For full enterprise support I suggest you refactor the group_vars and remove the prompts moving to full Ansible Vault – but for portability and ease of start-up I’ve made them prompted playbooks for now.

I would love to hear how they work out for you – please comment below if you have success!

Cisco Services APIs Ansible Playbooks – Version 2.0

I am very pleased to release the Cisco Services API Ansible Playbooks Version 2.0 which has been approved and released on Cisco DevNet Code Exchange !

You can find the code here and here

This major revision basically shifts away from lineinfile to Jinaj2 Templates for scale, performance, readability, and general best practices.

Serial 2 Info

The Cisco Serial 2 Info API receives a valid serial number and then returns structured JSON with your Cisco Contractual information !

The playbook uses the Genie parser to parse the show inventory command

After authenticating against the OAuth 2 service to get a Bearer token

It provides the API the serial number for every part per device.

The API provides the following information back:

Which we first dump into JSON and YAML files

Then template into CSV and MD

Using Jinja2

Which gives us:

Recommended Release

The other, very similar, Ansible playbook uses the Cisco Recommended Release API to create a spreadsheet with the current image on a host and the Cisco recommended version for that host given the Part ID (PID)

Here we don’t even have to use Genie to parse we can use the Ansible Facts module

And we transform again with Jinja2

And get this create report!

Please reach out to me directly if you need any help implementing these playbooks but I believe the instructions and code to be easy enough any beginner, with a little bit of refactoring and thought, could use this code as a starting point in their automation journey.

Dark Mode

Cisco_API_v2 (this link opens in a new window) by automateyournetwork (this link opens in a new window)

Ansible playbooks that capture serial number and PID and send them to the Cisco.com APIs transforming the response into business-ready documents. Version 2.0 uses Jinja2 templates.

A Recipe For Success – Using Ansible to Transform Cisco Data Centre Facts into Business-Ready Documentation

One of my favourite recipes is the Hakuna Frittata both because not only am I a big fan of puns, I also enjoy this hearty vegetarian meal that even I can handle putting together.

Inspired by this simple recipe I have decided to try and document my highly successful Ansible Cisco NXOS Facts playbook that captures and transforms raw facts from the data centre into business-ready documentation – automatically.

Ansible Cisco NXOS Facts to Business-Ready Documentation
Prep: 60-90 Min
Cook: 2-3 Min
Serves: An entire enterprise

Ingredients

1 Preheated Visual Studio Code
1 Git repository and Git
1 stick of Linux (a host with Ansible installed and SSH connectivity to the network devices)
3 pinches of Python filters
1 Cup of Ansible playbook (a YAML file with the serially executed tasks Ansible will perform)
1 Cup of Ansible module – NXOS_Facts
2 Tablespoons of Jinja2 Template
1 Teaspoon of hosts file
1 Tablespoon of group_vars
2 Raw Eggs – Cisco NXOS 7000 Aggregation Switches

Helpful Tip

This is not magic but did not necessarily come easy to me. You can use debug and print msg to yourself at the CLI. At each step that I register or have data inside a new variable I like to print it to the screen (one to see what the data, in JSON format, looks like; and two, to confirm my variable is not empty!)

Directions

1. You will need to first setup a hosts file listing your targeted hosts. I like to have a hierarchy as such:

hosts
[DC:children]
DCAgg
DCAccess

[DCAgg]
N7K01
N7K02

[DCAccess]
N5KA01
N5KB01
N5KA02
N5KB02

Or whatever your logical topology resembles.

2. Next we need to be able to securely connect to the devices. Create a group_vars folder and inside create a file that matches your hosts group name – in this case DC.yml

DC.yml
+

3. Create all the various output folder structure you require to store the files the playbook creates. I like something hierarchical again:

4. Create a playbooks folder to store the YAML file format Ansible playbook and a file called CiscoDCAggFacts.yml

In this playbook, which runs serially, we first capture the facts then transform them into business-ready documentation.

First we scope our targeted hosts (hosts: DCAgg)

Then we use the NXOS_Facts module to go gather all of the data. I want all the data so I choose gather_subset : – all but I could pick a smaller subset of facts to collect.

Next, and this is an important step, we take the captured data, now stored in the magic Ansible variable – {{ ansible_facts }} and put that into output files.

Using the | to_nice_json and | to_nice_yaml Python filters we can make the “RAW JSON” inside the variable (one long string if you were to look at it) into human-readable documentation.

4b. Repeatable step

NXOS Facts provides facts that can be put into the following distinct reports:

Platform information (hostname, serial number, license, software version, disk and memory information)
A list of all of the installed Modules hosted on the platform
A list of all IP addresses hosted on the platform
A list of all VLANs hosted on the platform
A list of all of the enabled Features on the platform
A list of all of the Interfaces, physical and virtual, including Fabric Extenders (FEX)
A list of all connected Neighbors
Fan information
Power Supply information

For some of these files, if the JSON data is structured in way that lends itself, I will create both a Comma-Separated Values (csv; a spreadsheet) file and a markdown (md; “html-light”) file. Some of the reports is just the csv file (IPs, Features, VLANs specifically).

The follow code can be copied 9 times and adjusted by updating the references – the task name, the template name, and the output file name – otherwise the basic structure is repeatable.

In order to create the HTML mind map you will also need mark map installed.

Another example of the code – this is the Interfaces section – notice only the name, src, and dest file names need to be updated as well as the MD and HTML file names in the shell command.

5. The Jinja2 Templates

Now that we have finished our Ansible playbook we need to create the Jinja2 templates we reference in the Ansible template module (in the src line)

Create the following folder structure to store the templates:

roles\dc\dc_agg\templates

Then, for each of the 9 templating tasks, create a matching .j2 file – for example the “base facts” as I like to call them – CiscoDCAggFacts.j2

In this template we need an If Else End If structure to test if we are templating csv or markdown then some For Loops to iterate over the JSON lists and key value pairs.

Add a header row with columns for the various fields of data. Reference your Nice JSON file to find the key value pairs.

No “For Loop” is required here just straight data from the JSON

Since its not csv it must be md; so add the appropriate markdown header rows

Then add the data row using markdown pipes for delimiters instead of commas

Close out the If

An example with For Loops might be Interfaces or Neighbors but the rest of the syntax and structure is the same

Now because there are multiple interfaces I need to loop or iterate over each interface.

Now add the row of data

Note you can include “In-line” If statements to check if a variable is defined. Some interfaces might not have a Description for example. Test if it is defined first, and if not (else) use a default of “No Description”

Other fields are imperative and do not need to be tested.

Close the Loop

Now do the markdown headers for Interfaces

Then the For Loop again and data row again but using pipes

Then close out the If statement

Complete the remaining templates. Save everything and Git commit / push up to your repo.

Cooking Time

Lets run the playbook against two fully-loaded production Nexus 7000s using the Linux time command

Two minutes in the oven !

Results

Some samples of the output.

First the Nice JSON – note the lists have been collapsed to be brief but any of the lists can be expanded in VS Code for the details

Interfaces

Neighbors

Now some prefer YAML to JSON so we have the exact same data but in YAML format as well

Now the above is already incredible but I wouldn’t call JSON and YAML files “business-ready” – for that we need a spreadsheet!

The real tasty stuff are the CSV files!

The general facts

Interfaces

Note that you can filter these csv files directly in VS Code – here I have applied a filter on all interfaces without a description

This captures all types of interfaces

Including SVIs

The Markdown provides a quickly rendered VS Code or browser experience

And the Interactive HTML is pretty neat!

Now remember we have all of these file types for all of the various facts these are just a few samples I like to hand out to the audience – for the full blown experience you can hopefully follow this recipe and cook your own Cisco NXOS Ansible Facts playbook!

This image has an empty alt attribute; its file name is image-72.png

Please reach out if you need any additional tips or advice ! I can be reached here or on my social media platforms.

Collaboration is the key to automation success – A Genie Success Story

I’ve been on about a three year journey with network automation and while I have had great personal and technical success – my organization and most of those outside my immediate day-to-day circle are still shackled to the ‘old’ way of doing things (primarily the CLI).

This year I decided to start a new program – a training session for those outside of my small development team – primarily targeted at the “operations” staff who can benefit the most from automation and infrastructure as code. This includes network operators, monitoring / NOC team members, IT Security staff, other developers, compute (server / storage) teams; everybody is welcome. We divided the calendar up into different teams with different recurring timeslots.

In advance I had written and tested a bunch of Code for the Campus Core – Ansible Facts and Genie parsed show commands – transforming the output into business-ready documentation. My plan was simple enough:

1. Ensure everybody was on the same page and had the same toolkit
* VS Code
* Git
* Azure DevOps repository bookmarked
* Various VS Code extensions
* A basic overview of main vs working branches in Git
* A basic outline on Ansible, YAML, Jinja2, and JSON

2. An operator would create a working development branch – in our case the Distribution Layer – so dist_facts Git branch.
* This operator ‘drives’ the whole session sharing their screen
* Step by step, line by line, refactoring (fancy way of saying copy-pasting) working code from the Core and updating it for the Distribution Layer as necessary
* Git clone, add, commit, push, and pull performed in both VS Code and Linux CLI
* Ansible playbook executed with a –limit against one building, then at scale after validating output
* Thorough tour of the JSON, YAML, CSV, MD, and HTML files after each run

3. Work through Ansible Facts and various Genie parsed commands to build up a source of truth

So far it has been a very successful approach with the two teams adopting Marvel superhero teams (TEAM: CAPTAIN AMERICA and TEAM: IRON MAN respectively) allowing me to create memes like this:

Image

Anyway – back to the point – today we had the following exchange:

Me: “So – we just parsed the show etherchannel summary CLI command and transformed the output into CSV files – amazing right! Any questions?”

Operator catching on quickly: “We use the show interfaces trunk command often to track down what VLANs are being carried on which interfaces – can we transform that into a CSV?”

Me, excited and proud of the Operator: “Amazing question and I’m glad you brought up a practical example of a command you use in the field all the time we can maybe transform into something a little more useful than CLI output!

Launch the Genie parser search engine (under available parsers on the left menu) and let’s see if there is a parser available

Bingo! Let’s do it!”

The playbook

In this example we are targeting the Campus Core.

The playbook is simple enough

Use the Ansible ios_command module to issue and register the show interfaces trunk command

Next, using the Genie filter plugin

Filter the raw variable and register a new variable with the Genie parsed results

Note you need to pass the parse_genie filter two arguments the command itself and the appropriate Cisco operating system

Next I like to create a Nice JSON and Nice YAML file with the parsed results as follows:

Which look like this:

And this:

I then use a loop to create a CSV and MD file from a Jinja2 template

The Jinaj2 Template looks like this

Couple things to hightlight:

* We challenge the current iteration of the Ansible loop and when it is on “csv” we template a CSV file format otherwise (else) it will be “md” and we template a markdown file
* The For Loop is over each interface in the results
* Be careful with the CSV file you need to regex_replace the comma out of results because they are comma-separated which will throw off your CSV file. The markdown does not require any regex.

Which results in this amazing sortable, searchable, filterable, version controller, source controller, truthful, fact-based CSV file:

Now this example is just the singular logical Core but we will quickly refactor the code next week in our next session together and the operator who wanted this playbook will get a chance to write it ! Then we will have the interface trunk information for the entire campus automatically in spreadsheets!

The moral of this story is to collaborate. Ask your front-line operators how automation can help them. Do they have any frequently used or highly valuable CLI Commands they want transformed into CSV format? Would Ansible facts help them? And then show them how to do it so they can start writing these playbooks for themselves.

Prevent an Intrusion – Run the Recommended Version!

In the wake of some very high profile IT security breaches and state sponsored attacks using compromised software today I wrote some infrastructure as code Ansible playbooks to create some business-ready documentation to help us understand our Cisco software version footprint against what release the vendor recommends. It is very important to run “Safe Harbor” code in the form of the Gold Star release. These releases are as close as it gets to being bug-free, secure, tested, and supported in production environments.

The ‘old-way’ involved getting the Cisco Part ID (PID) or several PIDs and looking up the recommended release on Cisco.com using an ever deepening hierarchy of platforms, operating systems, and PIDs. At scale this is like a day’s worth of work to go gather all of this information and present it in a way the business can understand.

Building on my recent success with the Serial2Info Cisco.com API as well as Ansible Facts I thought this might be another nice use-case for business-centric, non-technical (not routes, IP addresses, mac addresses, etc), extremely important and critical insight.

Use Case

Can I automatically get the PID from a host or group of hosts and provide it to the Cisco.com Software Suggestion API building business-ready reports in CSV and markdown?

Answer: Yes!

The Playbook

Again you are going to need:

* A Linux Host with SSH access to your Cisco IOS devices and HTTPS access to the Cisco.com API
* Credentials for the host and for the OAuth2 API
* We are not using Genie parsers here so just “base” Ansible will work

Step 1. Setup credential handling

Create a playbook file called CiscoCoreRecommendedReleaseFacts.yml

Again I use prompted methodology here same as the Serial2Info API

Gather the username, enable secret, Cisco.com API ClientID, Client Secret

Step 2. Gather Ansible Facts

Using the ios_facts module gather just the hardware subset

Because we are using Ansible Facts we do not need to register anything – the JSON is stored in the Ansible magic variable ansible_facts

I need 2 keys from this JSON – the PID and ideally the current running version. These can be found as follows in the ansible_facts variable:

Which is accessed as ansible_facts.net_model

Which again is accessed as ansible_facts.net_version

With the information above – without going any further – I could already build a nice report about what platforms and running versions there are!

But let’s go a step further and find out what Cisco recommends I should be running!

Step 2. Get your OAuth2 token

First, using the Ansible URI module

We need to get our token using the registered prompted credentials.

The API requires the following headers and body formatting; register the response as a variable (token):

We have to break apart the RAW JSON token to pass it to the ultimate Recommended Release API:

Now we are ready to send PIDs to the API.

Step 3 – Send PID to Cisco.com API

Again using the URI module:

Here we pass the ansible_facts.net_model Fact to the API as an HTTP GET:

The headers and body requirements. Notice the authentication and how we pass the Bearer Token along. We also register the returned JSON:

Here is what the returned JSON looks like:

The highest level key is json or accessed via RecommendedRelease.json

There is a productlist

Which as you can see is a list as denoted by the [ ]

Inside this list is another product key with the values from the API about the product itself

A little further down we find the recommended software release

Step 4 – Transform technical documentation into business ready CSV / MD files

These JSON and YAML (I also use the | to_nice_yaml filter to create a YAML file along with the JSON file) files are create for technical purposes but we can do a bit better making the information more palatable using business formats like CSV and mark down.

It is just a matter of using Jinja2 to template the CSV and Markdown files from the structured JSON variables / key-value pairs.

Add a final task in the Ansible playbook that will loop over the CSV and MD file types using the template module to source a new .j2 file – CiscoCoreRecommendedReleaseFacts.j2 – where our logic will go to generate our artifacts.

The Jinja2 starts with an If Else EndIf statement that checks if the Ansible loop is on CSV or not. If it is it uses the CSV section of templated file format otherwise it uses markdown syntax.

First we want to add a CSV header row

Then we need a For Loop to loop over each product in the productList

Now we add our “data line” per product in the loop using the various keys

Hostname for example uses the Ansible magic variable inventory_hostname

Then we want the Base PID. We use the Ansible default filter to set a default value in case the variable happens to be empty.

We continue accessing our keys and then we close the loop.

Now we need to create the Markdown syntax

And the same logic for the “data row” but with pipes instead of commas. Make sure to close off the If statement

Step 5 – Run playbook and check results

We run the playbook as ansible-playbook CiscoCoreRecommendedReleaseFacts.yml

Answer the prompts

Let the playbook run and check the results!

Summary

Again with a few free, simple tools like Ansible and the Cisco.com API we can, at scale gather and report on the current running version and the vendor recommended version quickly and easily and fully automatically!

Now go and start protecting your enterprise network armed with these facts!

Ansible Performance – Moving to Jinja2 for Automated Documentation

When Ivan Pepelnjak has advice for you – take it!

I wrote a post about untangling dynamic nested loops in Ansible.

In another recent post about trying to improve Ansible performance I didn’t get very far – but this could be the silver bullet I’ve been looking for to both optimize and make my Fact / Genie parsing playbooks more elegant code but also to bring my run times down so I can bring this from the lab to production.

Jinja2 Templates

One of the reasons why I perked up at Ivan’s generous suggestion is because I am a big fan and heavy user of Jinja2 templates already to generate intended configurations (Cisco IOS, NXOS configurations; JSON files for API POST) and documentation (intended configs in CSV, markdown, and HTML) – but I had just never thought of implementing them to create my documentation from received data!

My old way involved taking the structured JSON and using lineinfile or copy to create my output files. This was slow. Very slow.

Copy method:

Line In File method:

How to refactor this?

So I already have everything I need content wise – a header row and the data rows – I just need to move this into Jinja2 format. As it turns out there are some added benefits beyond just performance that I will highlight.

My quick use case was my CiscoNXOSFacts.yml playbook against 2 7Ks just gathering facts (nxos_facts) and transforming the structured JSON into business documentation.

– Create Nice JSON file from facts – Ansible | to_nice_json filter
– Create Nice YAML file from facts – Ansible | to_nice_yaml filter
– Create CSV file from facts
– Create markdown file from facts
– Generate HTML from markdown

So the first refactoring is the actual task from using copy or lineinfile to using template. Template needs a source (a new Jinja2 template file we will create in our next step).

Template also needs a destination. Here is where we can use the programmatic capabilities of Jinaj2 to simplify, optimize, and massively improve performance by setting up a simple loop and create both files. Wait files plural? Yes. My old way involved creating 2 separate files in 2 separate tasks. Now that I am using Jinja I can use variables – one item being “csv” and the other item being “md” – and pass them to the template for processing.

So create a Jinja2 template file called CiscoNXOSFactsTemplate.j2 to create your CSV and Markdown files.

Before I show the template I want to highlight another massive improvement to using Jinja2 – Jinaj2 is able to iterate naturally over dictionaries while my previous method had to pass the structured JSON through the | dict2items Ansible filter (against adding processing time). This simplifies the code quite a bit.

In the template we will test if the loop is on csv or md and create either a csv or md formatted output file.

Else if item is md create the markdown file format

One last and very important comment and benefit of Jinja2 is that I do not need to use Regular Expressions “as much” to clean up the JSON. | dict2items leaves a lot of garbage JSON characters behind which I had to previously use processor intensive RegEx tasks to clean up. Now Jinja2 does this cleanup and conversion from RAW to Nice JSON for me!

Results

I have only tested 1 playbook but I am very excited about this new refactored code !

Again this playbook “only” touches 2 physical devices but I have playbooks that potentially could be gathering facts and generating artifacts for hundreds of devices. But the results are pretty clear particularly the system time

Old way:

New way:

So roughly half the “real” time but look at the system time – from 36 seconds down a third to 12 seconds! WOW!

Thanks again!

A big thanks to Ivan for taking the time to comment and point me in a better direction. You may not know this but when I started my automation journey one of my resources along with several books, Cisco DevNet, trial and error, was my IPSpace.net subscription. If you are looking for a very affordable and very comprehensive library of networking and automation knowledge this is a good place to start.

Dynamic Loops in Ansible

I’ve done many great things with Ansible but occasionally I come across a logical problem that may stretch the tool past it’s limitations. If you have been following the site I am on a big facts discovery and automated documentation movement right now using Ansible Facts and Genie parsers.

The latest parser I am trying to convert to documentation is the show access-lists command.

So we use the Ansible ios_command module and issue the command then parse the response. This is a playbook called CiscoACLFacts.yml

I always start with creating Nice JSON and Nice YAML from the structured JSON returned by the parsed command:

Then I examine the Nice JSON to identify “how” I can parse the parsed data into business documentation in the form of CSV, markdown, and HTML files.

And I have my CLI command converted into amazing structured human or machine readable JSON:

So here is where the logical challenge comes into play with Ansible.

As you know an Access Control List (ACL) is a list (hint: it’s right in the name) of Access Control Entries (ACEs) which are also a list. Both the ACL and the ACE are variable – they could be almost anything – and I need 2 loops: an outer loop to iterate over the ACLs (shown above is the “1”) and an inner loop to iterate over the ACEs (shown above as the “10”).

So I brush up on my Ansible loops. Looks like I need with_nested.

Problem: Ansible does not support dynamic loops as you would expect. Here is what I “wanted to do” and tried for a while before I figured out it wasn’t supported:

with_nested:
– “{{ pyats_all_access_lists | dict2items }}”
– “{{ item[0].value.aces }}”

No go. “Item not found” errors.

Here is the work around / solution:

Before I get into the loops a couple things to point out to anyone new to infrastructure to code or JSON specifically. The Genie parsed return data is not a list by default. Meaning it cannot be iterated over with a loop. We have to filter this from a dictionary – as indicated in JSON by the { } delimiters – into a list (which would be indicated by [ ] delimiters in the JSON if it was a list we could iterate over) – before we can loop over it.

| dict2items is this filter.

The loops:

You can define the outer loop key using loop_var as part of loop_control along with include to build a dynamic outer / inner loop connection.

In order to create my CSV file:

1 – Delete the file outside the loops / last Ansible task before entering the loops

2 – * Important new step here *

We need to perform the outer loop, register the key for this outloop, and then include a separate YAML file that includes the inner loop task

3 – * Another important new step here *

Create the referenced file CiscoACLInnerLoop.yml with the inner loop task, in this case, the task to add the rows of data to the CSV file

Things to identify in the above task:

The loop – it is using the outer loop (the loop_var) ACL_list as the primary key then we turn the .value.aces dictionary into another list with | dict2items giving us the inner list we can iterate over.

Important – the inner loop is what Ansible will reference from this point forward meaning item. now references the inner items. In order to reference the outer key you need to reference the loop_var again as seen on the line: “{{ ACL_list.key }},{{ item.key }}

This gives us the ACL then the individual ACE per row in the CSV file! Mixing the outer and inner loops!

Recommendation – you will notice the start of an {% if %} {% else %} {% endif %} statement – because almost everything in an ACL and ACE list is variable you should test if each item.value.X is defined first, use the value if its defined, otherwise use a hard coded value. As such:

{% if item.value.logging is defined %}

{{ item.value.logging }}

{% else %}

No Logging

{% end if %}

Next, back in the main playbook file, outside the loop, we finally add our CSV header row:

For the sake of keeping this short there is likely some Regular Expression replacements we need to make to clean up any stray JSON or to remove unnecessary characters / strings left behind but in essence we have the follow show access-lists command rendered into CSV:

Operations, management, compliance and standard, and most of all IT SECURITY is going to love this! All of this is in a central Git repository so all of these artifacts are Git-tracked / enabled. All of the CSV files are searchable, sortable, filterable, and EXCEL ready for the business!

Summary

Before you give up on any problem make sure you find and read the documentation!

I have to revisit some previous use cases and problems now with fresh eyes and new capabilities because I had given up on transforming some previous dictionaries in dictionaries because I didn’t know what I was doing!

One step closer! I hope this article helped show you how dynamic Ansible looping is done and you don’t have to fail and struggle with the concept like I did. I am out there on Twitter if you have any questions!

Can we make Ansible go faster?

When I describe Ansible to people I tend to use many positive adjectives. Amazing, incredible, easy, revolutionary, powerful, and a few others. One adjective I never use, however, is fast. I would not describe Ansible as a high performance tool. Compared to manually doing the things I’ve come to automate with Ansible there is still no doubt I am saving hours if not days of effort. But now that I’m using Ansible for almost everything and at scale it would be great if I could get better performance out of the tool.

Over the years I’ve learned to run the ansible-playbook command then – and chant it like the late night informercial – “Set it and forget it!”

Its the one, sometimes painful, drawback I can find with Ansible. There has yet to be an infrastructure problem I have not been able to solve with Ansible – provided I am comfortable with waiting. “How long does this take?” change managers or operations will ask. “A while.” Is usually as optimistic as I can be.

(Note: It is a bit ironic sometimes the same crowd with complaints about how long a playbook takes to run are usually the same people who were comfortable with pre-automation manual-at-the-cli-of-every-device execution times into the days or weeks. Now anything more than a 10 minute playbook run seems like a long time. Go figure)

TL:DR

– Ansible is an amazing automation tool
– Ansible is not known for its performance
– Three modifications tried to make it go faster
– LibSSH
– Forks
– Pipelines
– No real improvements found with any of the above

Moving to LibSSH

The driving factor for me to bring my Ansible ecosystem into the shop and put it up on the lift to get underneath and into the mechanics of the configurations is this latest official blog post from Ansible.

“Not only is the new LibSSH connection plugin enabling FIPS readiness, but it was also designed to be more performant than the existing Paramiko SSH subsystem.”

This particular section of the blog post is what drives my exploration today. And yes FIPS readiness is important to me – the hook for me here is “designed to be more performant” – and yes the link they provide is great but I want to take the Pepsi Challenge myself.

Playbooks tested

I will be using the following playbooks with 2 different scale sets.

Cisco IOS Facts – Against my Lab distribution layer (4, Cisco 4500s) and my access layer (about 20 – 25 devices of various Catalyst flavours (2960, 3560, 3750, 3850, 9300)).

Cisco NXOS Facts – Same idea but against NXOS. 2 Nexus 7000 and 2 Nexus 5000.

The above playbooks use the Ansible facts modules. Let’s do some Genie Parsing of a show command as well.

IOS Genie show ip interface status

Methodology

In Linux you can use the time keyword command and prepend any command. Linux then provides three different timer – the real time, the user time, and the system time – results showing how long the command took to execute.

Result Set #1 – Defaults

With no changes to default Ansible here are the results. I will be standardizing on the sys results because of the input and other factors the real times and users times may have deviations:

Install LibSSH and modify Ansible

First step is to pip install the Ansible library we need:

Then we to update our persistent connections:

Refresh your Git repo and re-run the playbooks.

Result Set #2 – LibSSH

This image has an empty alt attribute; its file name is image-72.png
This image has an empty alt attribute; its file name is image-74.png
This image has an empty alt attribute; its file name is image-75.png
This image has an empty alt attribute; its file name is image-77.png
This image has an empty alt attribute; its file name is image-81.png

Forks

Ansible can be set to fork which allows multiple independent remote connections simultaneously.

Forks are intelligent and will only fork for the maximum number of remote targets. So I will set my forks in ansible.cfg to 50.

Now I don’t think this will help playbooks with under 5 targets because I believe Ansible defaults to 5 forks but maybe this will improve the Access Layer Facts which targets around 25 hosts. So lets just test against that one playbook.

Results Set #3 – Forks

This image has an empty alt attribute; its file name is image-74.png

Pipelining

Enabling pipelining reduces the number of SSH operations required to execute a module on the remote server, by executing many ansible modules without actual file transfer. According to the documentation this can result in a very significant performance improvement when enabled.

Add pipelining=true to the SSH connection section in ansible.cfg:

Result Set #4 – Pipelining

This image has an empty alt attribute; its file name is image-72.png
This image has an empty alt attribute; its file name is image-74.png
This image has an empty alt attribute; its file name is image-75.png
This image has an empty alt attribute; its file name is image-77.png
This image has an empty alt attribute; its file name is image-81.png

Summary

I didn’t have much success making Ansible go any faster either with the new LibSSH library, with forking, or with pipelining. I absolutely love ansible but the above times are from my small scale lab – imagine these run times in production at 5-10x the scale.

Have you found a way to make Ansible go faster that I’ve overlooked? Drop my a line!

Sean also jumped in to mention the driver for LibSSH was FIPS not performance and there are some performance improvements coming soon! Great!

Excitement

And he’s right! I can’t stop making new GitHub repositories with Genie parsed show commands to documentation! Like show ip interface brief as seen above!