Dynamic Loops in Ansible

I’ve done many great things with Ansible but occasionally I come across a logical problem that may stretch the tool past it’s limitations. If you have been following the site I am on a big facts discovery and automated documentation movement right now using Ansible Facts and Genie parsers.

The latest parser I am trying to convert to documentation is the show access-lists command.

So we use the Ansible ios_command module and issue the command then parse the response. This is a playbook called CiscoACLFacts.yml

I always start with creating Nice JSON and Nice YAML from the structured JSON returned by the parsed command:

Then I examine the Nice JSON to identify “how” I can parse the parsed data into business documentation in the form of CSV, markdown, and HTML files.

And I have my CLI command converted into amazing structured human or machine readable JSON:

So here is where the logical challenge comes into play with Ansible.

As you know an Access Control List (ACL) is a list (hint: it’s right in the name) of Access Control Entries (ACEs) which are also a list. Both the ACL and the ACE are variable – they could be almost anything – and I need 2 loops: an outer loop to iterate over the ACLs (shown above is the “1”) and an inner loop to iterate over the ACEs (shown above as the “10”).

So I brush up on my Ansible loops. Looks like I need with_nested.

Problem: Ansible does not support dynamic loops as you would expect. Here is what I “wanted to do” and tried for a while before I figured out it wasn’t supported:

with_nested:
– “{{ pyats_all_access_lists | dict2items }}”
– “{{ item[0].value.aces }}”

No go. “Item not found” errors.

Here is the work around / solution:

Before I get into the loops a couple things to point out to anyone new to infrastructure to code or JSON specifically. The Genie parsed return data is not a list by default. Meaning it cannot be iterated over with a loop. We have to filter this from a dictionary – as indicated in JSON by the { } delimiters – into a list (which would be indicated by [ ] delimiters in the JSON if it was a list we could iterate over) – before we can loop over it.

| dict2items is this filter.

The loops:

You can define the outer loop key using loop_var as part of loop_control along with include to build a dynamic outer / inner loop connection.

In order to create my CSV file:

1 – Delete the file outside the loops / last Ansible task before entering the loops

2 – * Important new step here *

We need to perform the outer loop, register the key for this outloop, and then include a separate YAML file that includes the inner loop task

3 – * Another important new step here *

Create the referenced file CiscoACLInnerLoop.yml with the inner loop task, in this case, the task to add the rows of data to the CSV file

Things to identify in the above task:

The loop – it is using the outer loop (the loop_var) ACL_list as the primary key then we turn the .value.aces dictionary into another list with | dict2items giving us the inner list we can iterate over.

Important – the inner loop is what Ansible will reference from this point forward meaning item. now references the inner items. In order to reference the outer key you need to reference the loop_var again as seen on the line: “{{ ACL_list.key }},{{ item.key }}

This gives us the ACL then the individual ACE per row in the CSV file! Mixing the outer and inner loops!

Recommendation – you will notice the start of an {% if %} {% else %} {% endif %} statement – because almost everything in an ACL and ACE list is variable you should test if each item.value.X is defined first, use the value if its defined, otherwise use a hard coded value. As such:

{% if item.value.logging is defined %}

{{ item.value.logging }}

{% else %}

No Logging

{% end if %}

Next, back in the main playbook file, outside the loop, we finally add our CSV header row:

For the sake of keeping this short there is likely some Regular Expression replacements we need to make to clean up any stray JSON or to remove unnecessary characters / strings left behind but in essence we have the follow show access-lists command rendered into CSV:

Operations, management, compliance and standard, and most of all IT SECURITY is going to love this! All of this is in a central Git repository so all of these artifacts are Git-tracked / enabled. All of the CSV files are searchable, sortable, filterable, and EXCEL ready for the business!

Summary

Before you give up on any problem make sure you find and read the documentation!

I have to revisit some previous use cases and problems now with fresh eyes and new capabilities because I had given up on transforming some previous dictionaries in dictionaries because I didn’t know what I was doing!

One step closer! I hope this article helped show you how dynamic Ansible looping is done and you don’t have to fail and struggle with the concept like I did. I am out there on Twitter if you have any questions!

Excitement

And he’s right! I can’t stop making new GitHub repositories with Genie parsed show commands to documentation! Like show ip interface brief as seen above!

Letting the Genie out of the bottle

Imagine if you could transform that unstructured Cisco show command screen output into something more useful than just CLI output.

What if there was a way to transform an SSH CLI show command’s response into a RESTful API-like HTTP GET request response in RESTful API-like structured JSON?

Sounds amazing right? Well with Genie this is totally possible. I mentioned the CTAS Genie / pyATS / xPresso solution in My Toolkit post. I also suggested that gathering facts with Ansible is the best place to start with network automation and infrastructure as code.

But the Ansible facts, while impressive, rich, plentiful, and extremely useful, they do not necessarily contain all of the state information that IOS / NXOS CLI show commands provide. Some information, such as CDP neighbors, interfaces, IP addresses, is available with only the ios_facts / nxos_facts modules but for things like the configured Virtual Route Forwarders (vrf) on a router, the IP Address Resolution Protocol (ARP) tables, or the OSPF routing tables you are stuck with crappy old Cisco CLI output right?

Wrong. You now have a magical Genie on your side who is here to grant all your state capture and transformation wishes! And you get more than 3!

TL;DR

– The historic restrictions of using Cisco IOS / NXOS show commands as CLI-only, raw screen / putty logged output, have been lifted.
– Genie parsers provide REST API HTTP GET-like responses to common CLI show commands.
– Ansible integrated allowing for running and parsing show commands at scale.
– I like to create RAW JSON, Nice JSON, Nice YAML, CSV, Markdown, and interactive HTML mind maps from any JSON I can get my hands on. Now I can do it with Cisco show commands!
– Fill the gaps from what is missing from base Ansible facts.
– Build a powerful, dynamic, state aware documentation set for every device in your network from every day show commands.
– Not as difficult as you might think.
– Another modern network automation, infrastructure as code, tool network engineers should include in their skillset.
– The best development team in the business. The Genie / pyATS / xPresso developers have personally helped me out. Find them on Cisco WebEx Teams for often real-time responses.

What is Genie?

Genie is a parser that automatically converts Cisco IOS / NXOS command output into structured JSON. This structured JSON allows developers to then work more programmatically with the output from important, but otherwise useless, show command output.

For example I am using Genie to parse some key show commands and create a dynamic automated library of documentation in different formats.

You can also go a step further with pyATS and write boolean tests (true / false) in Python using the Genie parsed data as your source of test data. The show ip ospf neighbor command, for example, can be executed, parsed with Genie, and then tested with pyATS! All of this can then be wrapped in business logic, scheduling, and protected with RBAC in xPresso.

Amazing but I am not made of money – how much does all this capability cost?

It is all free.

How do I integrate it with Ansible?

The amazing Ansible integration that I am using is thanks to Clay Curtis and his incredible contributions. Thanks to Clay there are two installation steps on top of the standard Ansible installation and an open Python filter plugin – then you are ready to go.

Please visit the Ansible Galaxy role, Cisco DevNet Code Exchange, and GitHub repository for all the details.

Show and Tell

It’s easier to just demonstrate how the Parser can be used with Ansible. Some prerequisites:

– Linux host
– pip install ansible
– pip install genie
– ansible-galaxy install clay584.parse_genie
– SSH access to network devices from this host
– Credentials for the device (prompted)
– The parse_genie Python filter_plugin
– Make sure your ansible.cfg specifies the location of the parse_genie file in filter_plugins.

[defaults]
filter_plugins=../filter_plugins

Example: show vrf

Recall what a show vrf looks like at the CLI:

This could spawn for pages depending on how many VRFs are hosted on the router. Also – how do you work with this output? Log to a putty file and inspect in notepad offline? Not very user friendly.

Let’s Genie parse that same command and see what the output looks like as structured JSON and take the Pepsi Challenge against the CLI.

In a playbook called CiscoVRFFacts.yml I first scope the playbook (CampusDistribution), then prompt for username and password. Note the collection includes Clay’s clay584.genie collection.

Next I run my standard Cisco show command with the ios_command module and register the response (which is RAW unparsed IOS config at this point) Nothing fancy here.

The next step is where we use the filter_plug in to parse the registered raw response and register a new variable that holds the parsed output. Again – this is not very complicated once you understand the syntax.

Note the parsed command is the same as the original command, in this case show vrf, and we have to specify the operating system (Cisco IOS).

You can optionally print these parsed facts, the nice JSON, to the screen.

Resulting in something like this:

We can save this output to a variety of files and manipulate the output in a variety of ways previously unavailable to us with raw standard CLI IOS output.

For starters lets put the RAW JSON response into a JSON file with the Ansible copy module.

Which is good for forensics, audits, security purposes, or for downstream systems that intake raw JSON, but it’s not very human readable.

Add the Ansible filter | to_nice_json to get the output to look like the output that is printed to the screen.

Now this is up for debate but personally I prefer and find YAML even more human-readable than JSON. Let’s make a YAML file with the | to_nice_yaml filter.

As a reminder this is what the show vrf command looks like at the CLI:

This image has an empty alt attribute; its file name is image-53.png

Now, in YAML:

Incredible!

Going a step further we can try to manipulate the output for business suitable reports in CSV, markdown, and HTML files.

Using yet another Ansible filter, the dict2items, which as the name implies transforms a dictionary to a list of items, we can loop over the original variable {{ pyats_all_vrfs.vrfs }} key and create our CSV / markdown.

(There are some Regular Expression (RegEx) steps that clean up the JSON a bit omitted for brevity)

Add a header row.

And now you have a CSV file!

Similar steps can create a markdown.

And then an HTML mind map can be generated.

Look at all the business and operational value we’ve squeezed out of a simple show vrf command!

All of this code is available on Automate Your Network’s GitHub.

Example: show ip arp

Start with the CLI show ip arp command output, which to be fair isn’t the worst CLI output around, which provides the ARP table for the Global Routing Table.

With more or less the same steps transform this into the same reports.

Setup the playbook:

Run the show ip arp command:

Parse it:

Create your 3 base RAW JSON / Nice JSON / Nice YAML files:

Check out this nice output!

Now anybody, CCNA level or not, can read the ordered structured list and see that VLAN20 has 1 neighbor with an IP of 172.24.2.1, the age, and the MAC address.

Similar steps to transform the output create the CSV / markdown / mind maps:

Also available on GitHub.

Example: show ip arp vrf {{ vrf }}

The exact same steps can be performs by simply adding show ip arp vrf <vrf name> with the same output as the Global Routing Table.

As a development aside I had big plans for show ip arp vrf {{ vrf }} to be a dynamic and automatically loop over all of the VRFs present on the route. I got pretty far but the parser itself hung me up.

Meaning I had a previous loop over the Genie parsed show vrf command which provided me the VRF name to feed the show ip arp vrf command. This all worked out and I could get the raw unparsed list like this:

ios_command:
commands:
– show ip arp vrf “{{ item.key }}”
loop: “{{ pyats_all_vrfs.vrf | dict2items }}”

But when it came time to parse this the following didn’t work.

| parse_genie(command=’show ip arp vrf {{ item.key }}’, os=’ios’)

I think because the parser is treating {{ item.key }} as raw text / the raw command and is not aware of the outer loop and to treat it like a variable. For the same reason I couldn’t write it to prompt for a VRF name. So, yes, I found one edge case drawback where I have to hardcode the VRF. Leave me a note if you see a way around this.

Summary

Genie parsers allow network engineers and operators to transform simple Cisco show commands into REST API-like requests with corresponding structured JSON response from the switch. This all magically happens behind the scenes allowing developers to actually make use of the show command output.

The days of setting up your Putty session to log your console to text files for offline analysis of raw standard output are finally over. Now you can work with JSON which in turn can be transformed into YAML, CSV, markdown, and HTML easily.

Along with Ansible facts, Genie parsed state data can solve the lifelong challenge of creating and maintaining good documentation.

Bonus code – show etherchannel summary

I couldn’t help myself and wrote another GitHub repository after finishing the blog. That’s how much I love Genie and how quick and easy it is!