Is Your Network At Risk – Automating the Cisco PSIRT with Genie and Ansible

The security of my network keeps me up at night. Honestly it does. We live in a world where enterprise networks are defending themselves against state-sponsored attacks. But what do attackers look for? Typically, open, well-known, vulnerabilities.

Well, at least to the hackers, attackers, and even script-kiddies, these vulnerabilities are well-known. Those playing defense are often a step or two behind just identifying vulnerabilities – and often times at the mercy of patch-management cycles or operational constraints that prevent them of addressing (patching) these waiting-to-be-exploited holes in the network.

What can we do about it?

The first thing we can do is to stay informed ! But this alone can be a difficult task with a fleet of various platforms running various software versions at scale. How many flavours of Cisco IOS, IOS-XE, and NXOS platforms make up your enterprise? What version are they running? And most importantly is that version compromised?

The old way might be to get e-mail notifications (hurray more e-mail!), maybe RSS-feeds, or go device-by-device, webpage-by-webpage, looking up the version and if it’s open to attack or not.

Do you see why, now, how an enterprise becomes vulnerable? It’s tedious and time-intensive work. And the moment you are done – the data is stale – what, are you going to wake up every day and review threats like this manually? Assign staff to do this? Just accept the risk and do they best you can trying to patch quarterly ?

Enter: Automation

These types of tasks beg to be solved with automation !

So how can you do it?

Let’s just lay out a high level, human language, defined use-case / wish list.

  1. Can we, at scale, go get the current IOS / IOS-XE / NXOS software version from a device?
  2. Then can we send that particular version somewhere to find out if it has been compromised ?
  3. Can we generate a report from the data above?

Technically the above is all feasible; easy even!

  1. Yes, we can use Ansible and the Cisco Genie Parser to capture the current software version
  2. The Cisco Product Security Incident Response Team has incredible, secure, REST APIs available that we can automate with the Ansible URI module
  3. Using Jinja2 templates we can craft business-ready reports

Getting Started

First you need to setup your Cisco.com REST API suite access

Pre-Ansible Development

As with any new REST API I like to start with Postman and then transform working requests into Ansible playbooks.

First, under a new or existing Cisco.com Collection, add a new request called IOS Vulnerabilities

Cisco.com uses OAuth2 authentication mechanism where you first must authenticate against one REST API (https://cloudsso.cisco.com/as/token.oauth2) which provides back authorization Bearer token used to then authenticate and authorize against subsequent Cisco.com APIs

Your Client ID and Secret are found in the API portal after you register the OpenVuln API

Request and use a token from the API in Postman:

Now add your request for IOS

https://api.cisco.com/security/advisories/ios?version=

Test it out!

Let’s hard code a version and see what flaws it has

Ok so it has at least 1 open vulnerability!

Does it tell us what version fixes it?

Ok let’s check that version quickly while we are still in Postman

This version has no disclosed vulnerabilities!

One interesting thing of note – and our automation is going to need to handle it – is that if there are no flaws found we get a 404 back from the API not a 200 like our flawed response!

The Playbook

For the sake of the example I am using prompted inputs / response but these variables could easily be hardcoded and Ansible Vaulted.

So first prompt for your Cisco hosts username and password and your Cisco.com ClientID and Client Secret

Register the response

Then in the IOS.yml group_vars I have my Ansible network connections

I’ve put all IOS-platforms in this group in the hosts file to target them

Next step run the show version ios_command and register the reponse

Genie parse and register the JSON

Now we need, just like in Postman, to go get the OAuth2 token but instead of Postman we need to use the Ansible URI module

Then we have to parse this response and setup our Bearer and Token components from the response JSON.

Now that we have our token we can authenticate against the IOS open Vulernability API

There are a couple of things going on here:

  1. We are passing the Genie parsed .version.version key to the API for each IOS host in our list
  2. We are using the {{ token_type }} and {{ access_token }} to Authorize
  3. We have to expect two different status codes; 200 (flaws found on a host) and 404 (no flaws for the host software version)
  4. I’ve added until and delay to slow down / throttle the API requests as not to get a 406 back because I’ve overwhelmed the 10 requests per second upper limit
  5. We register the JSON response from the API

As always I like to create a “nice” (easy to read) version of the output in a .json file

Note we need to Loop over each host in our playbook (using the Ansible magic variable ansible_play_hosts) so the JSON file has a data set for each host in the playbook.

Lastly we run the template module to pass the data into Jinja2 where we will create a business-ready CSV file

Which looks like this broken apart:

Like building the JSON file first we need to loop over the hosts in the playbook.

Then we check if the errorCode is defined. You could also look at the 404 status code here. Either way if you get the error it means there are no vulnerabilities.

So add a row of data with the hostname, “N/A” for most fields, and the json.errorMessage from the API (or just hardcode “No flaws” or whatever you want here; “compliant”)

Now if the errorCode is not defined it means there are open flaws and we will get back other data we need to loop into.

I am choosing to do a nested set of two loops – one for each advisory in the list of advisories per software version. Then inside that loop another loop, adding a row of data to the spreadsheet, for each BugID found as well (which is also a list).

There are a few more lists like CVEs for example – you can either loop of these as well (but we start to get into too much repetition / rows of data) – or just use regex_replace(‘,’,’ ‘) to remove all commas inside a field. The result is the list spaced out inside the cell sorted alphabetically but if you do not do this it will throw off the number of cells in your CSV

The results

What can you do with “just” a CSV file?

With Excel or simply VS Code with the Excel Preview extension – some pretty awesome things!

I can, for example, pick the ID field and filter down a particular flaw in the list – which would then provide me all the hosts affected by that bug

Or pick a host out of the list and see what flaws it has, if any. Or any of the columns we have setup in the Jinja2

This image has an empty alt attribute; its file name is image-108.png

Included in the report is also the SEVERITY and BASE SCORE for quick decision making or the Detailed Publication URL to get a detailed report for closer analysis.

Which looks like this:

Automation is much more than configuration management. It can be used to manage, understand, and mitigate risk as well.

Together we can secure our enterprise networks and hopefully sleep a little more sound knowing they are safe.

Cisco Facts – A Collection of Ansible IOS / NXOS Facts and Genie Parsing Playbooks

Cisco DevNet Code Exchange has published my repository !

Dark Mode

Cisco_Facts (this link opens in a new window) by automateyournetwork (this link opens in a new window)

Ansible playbooks that use the IOS / NXOS Facts modules and Genie parsed commands to transform RAW JSON into business-ready documentation

Here you can easily start capturing Ansible Facts for IOS and NXOS and transform the JSON into CSV and Markdown !

Also included are a bunch of valuable Genie parsed show commands which transform the response into JSON then again transforms the JSON into CSV and Markdown!

The playbooks use Prompts so you should be able to clone the repo and update the hosts file and start targeting your hosts! For full enterprise support I suggest you refactor the group_vars and remove the prompts moving to full Ansible Vault – but for portability and ease of start-up I’ve made them prompted playbooks for now.

I would love to hear how they work out for you – please comment below if you have success!

Cisco Services APIs Ansible Playbooks – Version 2.0

I am very pleased to release the Cisco Services API Ansible Playbooks Version 2.0 which has been approved and released on Cisco DevNet Code Exchange !

You can find the code here and here

This major revision basically shifts away from lineinfile to Jinaj2 Templates for scale, performance, readability, and general best practices.

Serial 2 Info

The Cisco Serial 2 Info API receives a valid serial number and then returns structured JSON with your Cisco Contractual information !

The playbook uses the Genie parser to parse the show inventory command

After authenticating against the OAuth 2 service to get a Bearer token

It provides the API the serial number for every part per device.

The API provides the following information back:

Which we first dump into JSON and YAML files

Then template into CSV and MD

Using Jinja2

Which gives us:

Recommended Release

The other, very similar, Ansible playbook uses the Cisco Recommended Release API to create a spreadsheet with the current image on a host and the Cisco recommended version for that host given the Part ID (PID)

Here we don’t even have to use Genie to parse we can use the Ansible Facts module

And we transform again with Jinja2

And get this create report!

Please reach out to me directly if you need any help implementing these playbooks but I believe the instructions and code to be easy enough any beginner, with a little bit of refactoring and thought, could use this code as a starting point in their automation journey.

Dark Mode

Cisco_API_v2 (this link opens in a new window) by automateyournetwork (this link opens in a new window)

Ansible playbooks that capture serial number and PID and send them to the Cisco.com APIs transforming the response into business-ready documents. Version 2.0 uses Jinja2 templates.

Collaboration is the key to automation success – A Genie Success Story

I’ve been on about a three year journey with network automation and while I have had great personal and technical success – my organization and most of those outside my immediate day-to-day circle are still shackled to the ‘old’ way of doing things (primarily the CLI).

This year I decided to start a new program – a training session for those outside of my small development team – primarily targeted at the “operations” staff who can benefit the most from automation and infrastructure as code. This includes network operators, monitoring / NOC team members, IT Security staff, other developers, compute (server / storage) teams; everybody is welcome. We divided the calendar up into different teams with different recurring timeslots.

In advance I had written and tested a bunch of Code for the Campus Core – Ansible Facts and Genie parsed show commands – transforming the output into business-ready documentation. My plan was simple enough:

1. Ensure everybody was on the same page and had the same toolkit
* VS Code
* Git
* Azure DevOps repository bookmarked
* Various VS Code extensions
* A basic overview of main vs working branches in Git
* A basic outline on Ansible, YAML, Jinja2, and JSON

2. An operator would create a working development branch – in our case the Distribution Layer – so dist_facts Git branch.
* This operator ‘drives’ the whole session sharing their screen
* Step by step, line by line, refactoring (fancy way of saying copy-pasting) working code from the Core and updating it for the Distribution Layer as necessary
* Git clone, add, commit, push, and pull performed in both VS Code and Linux CLI
* Ansible playbook executed with a –limit against one building, then at scale after validating output
* Thorough tour of the JSON, YAML, CSV, MD, and HTML files after each run

3. Work through Ansible Facts and various Genie parsed commands to build up a source of truth

So far it has been a very successful approach with the two teams adopting Marvel superhero teams (TEAM: CAPTAIN AMERICA and TEAM: IRON MAN respectively) allowing me to create memes like this:

Image

Anyway – back to the point – today we had the following exchange:

Me: “So – we just parsed the show etherchannel summary CLI command and transformed the output into CSV files – amazing right! Any questions?”

Operator catching on quickly: “We use the show interfaces trunk command often to track down what VLANs are being carried on which interfaces – can we transform that into a CSV?”

Me, excited and proud of the Operator: “Amazing question and I’m glad you brought up a practical example of a command you use in the field all the time we can maybe transform into something a little more useful than CLI output!

Launch the Genie parser search engine (under available parsers on the left menu) and let’s see if there is a parser available

Bingo! Let’s do it!”

The playbook

In this example we are targeting the Campus Core.

The playbook is simple enough

Use the Ansible ios_command module to issue and register the show interfaces trunk command

Next, using the Genie filter plugin

Filter the raw variable and register a new variable with the Genie parsed results

Note you need to pass the parse_genie filter two arguments the command itself and the appropriate Cisco operating system

Next I like to create a Nice JSON and Nice YAML file with the parsed results as follows:

Which look like this:

And this:

I then use a loop to create a CSV and MD file from a Jinja2 template

The Jinaj2 Template looks like this

Couple things to hightlight:

* We challenge the current iteration of the Ansible loop and when it is on “csv” we template a CSV file format otherwise (else) it will be “md” and we template a markdown file
* The For Loop is over each interface in the results
* Be careful with the CSV file you need to regex_replace the comma out of results because they are comma-separated which will throw off your CSV file. The markdown does not require any regex.

Which results in this amazing sortable, searchable, filterable, version controller, source controller, truthful, fact-based CSV file:

Now this example is just the singular logical Core but we will quickly refactor the code next week in our next session together and the operator who wanted this playbook will get a chance to write it ! Then we will have the interface trunk information for the entire campus automatically in spreadsheets!

The moral of this story is to collaborate. Ask your front-line operators how automation can help them. Do they have any frequently used or highly valuable CLI Commands they want transformed into CSV format? Would Ansible facts help them? And then show them how to do it so they can start writing these playbooks for themselves.

Sign on the Dotted Line

Layer 9 issues – finance – are often some of the most challenging a network engineer faces. Contract management can be particularly difficult in any scale organization especially if you are not “sole source” purchasing. Serial numbers and contracts are also not typically things the “network people” want to deal with but when that P1 hits and you try to open a SEV 1 TAC CASE – only to find out you are not under contract – I’ve been in less terrifying car accidents than this nightmare scenario.

I have good news ! Using a mix of automation and developer-like tools the network engineer can now create a real source of truth that, along with routes and MAC address-tables and other technical information, can include inventory and contractual business documentation from stateful, truthful, real-time, facts from Cisco.

Ok so let’s get into it!

As a rough outline for our logic here is the use case:

Can I automatically gather the serial numbers from Cisco device hostnames and then provide them to Cisco and get my contractual state for each part on each device?

Answer: YES !

What you will need:

* Linux host with Ansible, Genie parser
* Linux host requires both SSH access to the Cisco host and Internet Access to the OAuth2 and Cisco.com API HTTPS URLs
* Cisco SmartNet Total Care – I have written up instructions in this repo under the “OnBoarding Process” section

The Playbook

Step 1 – We will need to get the serial number for every part for a given hostname. For this we will use the standard show inventory command for IOS using the Ansible ios_command module. I will be using prompted methods for demonstration purposes or for on-demand multi-user (each with their own accounts) runtime, but we could easily Ansible Vault these credentials for fully hands-free run time or to containerize this playbook. I am also targeting a specific host – the Core – but I could easily change this to be every IOS device in the enterprise. This playbook is called CiscoCoreSerial2InfoFacts.yml

First prompt for username, enable secret, Cisco Customer ID, Cisco Customer Secret and register these variables:

Then run the ios_command show inventory and register the results in a variable.

Step 2 – Parse the raw output from the IOS command

Next, we use Genie to parse the raw results and register a new variable with the structured JSON. Genie requires, for show inventory, the command, the operating system, and the platform (in this case a Cisco 6500)

And here is what that structured JSON looks like:

So now we have a nice list of each part and their serial number we can feed the Cisco.com API to get back our contract information.

Step 3 – Get an OAuth 2 token from Cisco web services.

Cisco.com APIs use OAuth2 for authentication meaning you can not go directly against the API with a username and password. First you must retrieve a Bearer Token and then use that limited time token within it’s lifetime against the ultimate API.

Using the Ansible URI module go get a token and register the results as a variable. Provide the Customer ID and Client secret prompts to the API for authentication. This is an HTTP POST method.

With the new raw token setup the token type and access token from the raw response

Step 4 – Provide token to the Serial2Contract Cisco API to get back contractual information for each serial number.

Now that we have a valid token we can authenticate and authorize against the Cisco SmartNet Total Care Serial Number 2 Contract Information API.

In this step we are going to use an Ansible loop to loop over the Genie parsed structured JSON from the show inventory command providing the sn key for each item in the list. We need to use the Python | dict2items Ansible filter to transform the dictionary into a list we can iterate over.

The loop is written as

loop: “{{ pyats_inventory.index | dict2items }}”

And each serial number is referenced in the URL each iteration through the loop:

url: https://api.cisco.com/sn2info/v2/coverage/summary/serial_numbers/{{ item.value.sn }}

We register the returned structured JSON from the API as Serial2Info which looks like this:

So now I have the JSON – let’s make it a business ready artifact – a CSV file / spreadsheet and a markdown file – using Jinja2

Step 5 – Using Jinja2 lets template the structured JSON into a CSV file for the business.

Create a matching Jinja2 template called CiscoCoreSerial2InfoFacts.j2 and add a task to Ansible that uses the template module to build both a CSV file and a markdown file from the JSON.

In the Jinja2 file we need a section for CSV (if item = “csv”) and a section for markdown (else) based on their respective syntax. Then we need to loop over each of the responses.

result in Serial2Info[‘results’] is the loop used. I also add a default value using a filter | default (‘N/A’) in case the value is not defined. SFPs for example do not have all of the fields that a supervisor module has so to be safe it’s best to build in a default value for each variable.

The final Jinja2 looks something like this:

Which results in a CSV and Markdown file with a row for every serial number and their contractual facts from the API.

Summary

Large scale inventory and contract information can easily be automated into CSV spreadsheets that the business can easily consume. Ansible, Genie, Cisco.com APIs, Jinja2 templating and a little bit a logic come together into an automation pipeline that ensures contractual compliance and inventory fidelity at scale!

Dynamic Loops in Ansible

I’ve done many great things with Ansible but occasionally I come across a logical problem that may stretch the tool past it’s limitations. If you have been following the site I am on a big facts discovery and automated documentation movement right now using Ansible Facts and Genie parsers.

The latest parser I am trying to convert to documentation is the show access-lists command.

So we use the Ansible ios_command module and issue the command then parse the response. This is a playbook called CiscoACLFacts.yml

I always start with creating Nice JSON and Nice YAML from the structured JSON returned by the parsed command:

Then I examine the Nice JSON to identify “how” I can parse the parsed data into business documentation in the form of CSV, markdown, and HTML files.

And I have my CLI command converted into amazing structured human or machine readable JSON:

So here is where the logical challenge comes into play with Ansible.

As you know an Access Control List (ACL) is a list (hint: it’s right in the name) of Access Control Entries (ACEs) which are also a list. Both the ACL and the ACE are variable – they could be almost anything – and I need 2 loops: an outer loop to iterate over the ACLs (shown above is the “1”) and an inner loop to iterate over the ACEs (shown above as the “10”).

So I brush up on my Ansible loops. Looks like I need with_nested.

Problem: Ansible does not support dynamic loops as you would expect. Here is what I “wanted to do” and tried for a while before I figured out it wasn’t supported:

with_nested:
– “{{ pyats_all_access_lists | dict2items }}”
– “{{ item[0].value.aces }}”

No go. “Item not found” errors.

Here is the work around / solution:

Before I get into the loops a couple things to point out to anyone new to infrastructure to code or JSON specifically. The Genie parsed return data is not a list by default. Meaning it cannot be iterated over with a loop. We have to filter this from a dictionary – as indicated in JSON by the { } delimiters – into a list (which would be indicated by [ ] delimiters in the JSON if it was a list we could iterate over) – before we can loop over it.

| dict2items is this filter.

The loops:

You can define the outer loop key using loop_var as part of loop_control along with include to build a dynamic outer / inner loop connection.

In order to create my CSV file:

1 – Delete the file outside the loops / last Ansible task before entering the loops

2 – * Important new step here *

We need to perform the outer loop, register the key for this outloop, and then include a separate YAML file that includes the inner loop task

3 – * Another important new step here *

Create the referenced file CiscoACLInnerLoop.yml with the inner loop task, in this case, the task to add the rows of data to the CSV file

Things to identify in the above task:

The loop – it is using the outer loop (the loop_var) ACL_list as the primary key then we turn the .value.aces dictionary into another list with | dict2items giving us the inner list we can iterate over.

Important – the inner loop is what Ansible will reference from this point forward meaning item. now references the inner items. In order to reference the outer key you need to reference the loop_var again as seen on the line: “{{ ACL_list.key }},{{ item.key }}

This gives us the ACL then the individual ACE per row in the CSV file! Mixing the outer and inner loops!

Recommendation – you will notice the start of an {% if %} {% else %} {% endif %} statement – because almost everything in an ACL and ACE list is variable you should test if each item.value.X is defined first, use the value if its defined, otherwise use a hard coded value. As such:

{% if item.value.logging is defined %}

{{ item.value.logging }}

{% else %}

No Logging

{% end if %}

Next, back in the main playbook file, outside the loop, we finally add our CSV header row:

For the sake of keeping this short there is likely some Regular Expression replacements we need to make to clean up any stray JSON or to remove unnecessary characters / strings left behind but in essence we have the follow show access-lists command rendered into CSV:

Operations, management, compliance and standard, and most of all IT SECURITY is going to love this! All of this is in a central Git repository so all of these artifacts are Git-tracked / enabled. All of the CSV files are searchable, sortable, filterable, and EXCEL ready for the business!

Summary

Before you give up on any problem make sure you find and read the documentation!

I have to revisit some previous use cases and problems now with fresh eyes and new capabilities because I had given up on transforming some previous dictionaries in dictionaries because I didn’t know what I was doing!

One step closer! I hope this article helped show you how dynamic Ansible looping is done and you don’t have to fail and struggle with the concept like I did. I am out there on Twitter if you have any questions!

Excitement

And he’s right! I can’t stop making new GitHub repositories with Genie parsed show commands to documentation! Like show ip interface brief as seen above!

Letting the Genie out of the bottle

Imagine if you could transform that unstructured Cisco show command screen output into something more useful than just CLI output.

What if there was a way to transform an SSH CLI show command’s response into a RESTful API-like HTTP GET request response in RESTful API-like structured JSON?

Sounds amazing right? Well with Genie this is totally possible. I mentioned the CTAS Genie / pyATS / xPresso solution in My Toolkit post. I also suggested that gathering facts with Ansible is the best place to start with network automation and infrastructure as code.

But the Ansible facts, while impressive, rich, plentiful, and extremely useful, they do not necessarily contain all of the state information that IOS / NXOS CLI show commands provide. Some information, such as CDP neighbors, interfaces, IP addresses, is available with only the ios_facts / nxos_facts modules but for things like the configured Virtual Route Forwarders (vrf) on a router, the IP Address Resolution Protocol (ARP) tables, or the OSPF routing tables you are stuck with crappy old Cisco CLI output right?

Wrong. You now have a magical Genie on your side who is here to grant all your state capture and transformation wishes! And you get more than 3!

TL;DR

– The historic restrictions of using Cisco IOS / NXOS show commands as CLI-only, raw screen / putty logged output, have been lifted.
– Genie parsers provide REST API HTTP GET-like responses to common CLI show commands.
– Ansible integrated allowing for running and parsing show commands at scale.
– I like to create RAW JSON, Nice JSON, Nice YAML, CSV, Markdown, and interactive HTML mind maps from any JSON I can get my hands on. Now I can do it with Cisco show commands!
– Fill the gaps from what is missing from base Ansible facts.
– Build a powerful, dynamic, state aware documentation set for every device in your network from every day show commands.
– Not as difficult as you might think.
– Another modern network automation, infrastructure as code, tool network engineers should include in their skillset.
– The best development team in the business. The Genie / pyATS / xPresso developers have personally helped me out. Find them on Cisco WebEx Teams for often real-time responses.

What is Genie?

Genie is a parser that automatically converts Cisco IOS / NXOS command output into structured JSON. This structured JSON allows developers to then work more programmatically with the output from important, but otherwise useless, show command output.

For example I am using Genie to parse some key show commands and create a dynamic automated library of documentation in different formats.

You can also go a step further with pyATS and write boolean tests (true / false) in Python using the Genie parsed data as your source of test data. The show ip ospf neighbor command, for example, can be executed, parsed with Genie, and then tested with pyATS! All of this can then be wrapped in business logic, scheduling, and protected with RBAC in xPresso.

Amazing but I am not made of money – how much does all this capability cost?

It is all free.

How do I integrate it with Ansible?

The amazing Ansible integration that I am using is thanks to Clay Curtis and his incredible contributions. Thanks to Clay there are two installation steps on top of the standard Ansible installation and an open Python filter plugin – then you are ready to go.

Please visit the Ansible Galaxy role, Cisco DevNet Code Exchange, and GitHub repository for all the details.

Show and Tell

It’s easier to just demonstrate how the Parser can be used with Ansible. Some prerequisites:

– Linux host
– pip install ansible
– pip install genie
– ansible-galaxy install clay584.parse_genie
– SSH access to network devices from this host
– Credentials for the device (prompted)
– The parse_genie Python filter_plugin
– Make sure your ansible.cfg specifies the location of the parse_genie file in filter_plugins.

[defaults]
filter_plugins=../filter_plugins

Example: show vrf

Recall what a show vrf looks like at the CLI:

This could spawn for pages depending on how many VRFs are hosted on the router. Also – how do you work with this output? Log to a putty file and inspect in notepad offline? Not very user friendly.

Let’s Genie parse that same command and see what the output looks like as structured JSON and take the Pepsi Challenge against the CLI.

In a playbook called CiscoVRFFacts.yml I first scope the playbook (CampusDistribution), then prompt for username and password. Note the collection includes Clay’s clay584.genie collection.

Next I run my standard Cisco show command with the ios_command module and register the response (which is RAW unparsed IOS config at this point) Nothing fancy here.

The next step is where we use the filter_plug in to parse the registered raw response and register a new variable that holds the parsed output. Again – this is not very complicated once you understand the syntax.

Note the parsed command is the same as the original command, in this case show vrf, and we have to specify the operating system (Cisco IOS).

You can optionally print these parsed facts, the nice JSON, to the screen.

Resulting in something like this:

We can save this output to a variety of files and manipulate the output in a variety of ways previously unavailable to us with raw standard CLI IOS output.

For starters lets put the RAW JSON response into a JSON file with the Ansible copy module.

Which is good for forensics, audits, security purposes, or for downstream systems that intake raw JSON, but it’s not very human readable.

Add the Ansible filter | to_nice_json to get the output to look like the output that is printed to the screen.

Now this is up for debate but personally I prefer and find YAML even more human-readable than JSON. Let’s make a YAML file with the | to_nice_yaml filter.

As a reminder this is what the show vrf command looks like at the CLI:

This image has an empty alt attribute; its file name is image-53.png

Now, in YAML:

Incredible!

Going a step further we can try to manipulate the output for business suitable reports in CSV, markdown, and HTML files.

Using yet another Ansible filter, the dict2items, which as the name implies transforms a dictionary to a list of items, we can loop over the original variable {{ pyats_all_vrfs.vrfs }} key and create our CSV / markdown.

(There are some Regular Expression (RegEx) steps that clean up the JSON a bit omitted for brevity)

Add a header row.

And now you have a CSV file!

Similar steps can create a markdown.

And then an HTML mind map can be generated.

Look at all the business and operational value we’ve squeezed out of a simple show vrf command!

All of this code is available on Automate Your Network’s GitHub.

Example: show ip arp

Start with the CLI show ip arp command output, which to be fair isn’t the worst CLI output around, which provides the ARP table for the Global Routing Table.

With more or less the same steps transform this into the same reports.

Setup the playbook:

Run the show ip arp command:

Parse it:

Create your 3 base RAW JSON / Nice JSON / Nice YAML files:

Check out this nice output!

Now anybody, CCNA level or not, can read the ordered structured list and see that VLAN20 has 1 neighbor with an IP of 172.24.2.1, the age, and the MAC address.

Similar steps to transform the output create the CSV / markdown / mind maps:

Also available on GitHub.

Example: show ip arp vrf {{ vrf }}

The exact same steps can be performs by simply adding show ip arp vrf <vrf name> with the same output as the Global Routing Table.

As a development aside I had big plans for show ip arp vrf {{ vrf }} to be a dynamic and automatically loop over all of the VRFs present on the route. I got pretty far but the parser itself hung me up.

Meaning I had a previous loop over the Genie parsed show vrf command which provided me the VRF name to feed the show ip arp vrf command. This all worked out and I could get the raw unparsed list like this:

ios_command:
commands:
– show ip arp vrf “{{ item.key }}”
loop: “{{ pyats_all_vrfs.vrf | dict2items }}”

But when it came time to parse this the following didn’t work.

| parse_genie(command=’show ip arp vrf {{ item.key }}’, os=’ios’)

I think because the parser is treating {{ item.key }} as raw text / the raw command and is not aware of the outer loop and to treat it like a variable. For the same reason I couldn’t write it to prompt for a VRF name. So, yes, I found one edge case drawback where I have to hardcode the VRF. Leave me a note if you see a way around this.

Summary

Genie parsers allow network engineers and operators to transform simple Cisco show commands into REST API-like requests with corresponding structured JSON response from the switch. This all magically happens behind the scenes allowing developers to actually make use of the show command output.

The days of setting up your Putty session to log your console to text files for offline analysis of raw standard output are finally over. Now you can work with JSON which in turn can be transformed into YAML, CSV, markdown, and HTML easily.

Along with Ansible facts, Genie parsed state data can solve the lifelong challenge of creating and maintaining good documentation.

Bonus code – show etherchannel summary

I couldn’t help myself and wrote another GitHub repository after finishing the blog. That’s how much I love Genie and how quick and easy it is!

My current toolkit

“We become what we behold. We shape our tools, and thereafter our tools shape us.”
― Marshall McLuhan

My first attempt at network automation I used the tools that I had used to manage infrastructure with for the past 20+ years. A file editor (notepad) and a file transfer program (WinSCP) on my Windows 10 machine and a Linux (CentOS (at the office) / Ubuntu (at home)) machine. Ansible was the first new tool introduced to me and every other tool here has followed to either support my Ansible development or I was led down the path to discovery because of Ansible. So – installed Ansible on the Linux environment at the office and made sure it had SSH connectivity to my in-band management network.

Ready to go! I had everything I needed to write YAML files and make my playbooks, group / host variable files, and inventory file. I could either write these in Notepad in Windows and transfer them to Linux or just “vi” them directly on Linux. All set right?

As a beginner trying to orchestrate a series of serially executed commands in Ansible playbook tasks I was suffering from a mix of ignorance and arrogance. I didn’t know what I didn’t know. And while yes, my playbook would eventually go onto be successful and make a large scale, complex, change across multiple devices without causing an outage of any kind, the process was brutal. Back and forth trial and error with all of these, what turned out to be, unnecessary steps of trying to track my latest version of working code across my development environments.

Shameful filenames like “John_Working_Code_v2_latest_new02.yml” were sprawling out of control and I was starting to feel like a “The Price is Right” contestant where they guess the value of 5 items, pull a lever to see how many they got right, then run back and guess again and try to figure out which ones were correct, then run back and pull the lever again.

Eventually I got all 5 items priced correctly but it was a lot of panic-driven, chaotic, running around, as it turns out, for no reason.

Was there a better way? Surely this isn’t what people mean when they say DevOps or infrastructure as code or network automation. Why would anyone do it this way? It doesn’t scale. It took weeks longer than had I just logged into each router at the CLI and configured the device manually by hand.

Ansible wasn’t that hard – but the tools I was using were simply wrong. Enter the modern toolkit.

TL:DR

– This toolkit took about three years to put together through a lot of hard work and discovery.
– Tools make all the difference.
– You need an Integrated Development Environment (IDE).
– Version and source control are good things. They are not just middle management talk.
– Network automation means treating infrastructure as code.
– You have crossed over from IT into Development; act accordingly.
– Use software development tools to solve software development problems.
– Git. Git. Git. Git. Git. Git.
– Powerful stuff.
– Leads to Continuous Integration / Continuous Delivery (CI/CD) in the long run.
– Both for configuration management as well as state capture, validation, and testing.

Version / Source Control – Git

I start with Git because in order of software installation Git should come first. You want to build up a development environment in order to work with infrastructure as code and you want to have both version (current, working, code; previous working code; testing new things without affecting old working things) and source (source of truth, which copy is the master copy / primary copy / main copy, allow for distributed development) controls (like RBAC but over your code base). For Windows you need to download and install Git (first, before your IDE so Git can be integrated when you install your IDE). But for Linux Git comes pre-installed as a standard.

Git vs GitHub

Git is not GitHub and GitHub is not Git.

Git is the actual software that does the version and source control. Git creates a hidden .git folder that tracks changes inside that Git-enabled folder. Git has commands used to work with code.

GitHub is an online Git repository. The largest collection of code in the universe GitHub provides a free place to store Git repositories (the folder with the .git subfolder tracking all artifacts within the parent folder). GitHub, or other Git hosting repository sites or services, provides the source control over your infrastructure as code.

Git is used to clone Git repositories (GitHub or other Git repository hosting site) locally, that is take a full copy of the remote repository locally, where developers can make changes and then push those changes back into the remote repository.

Branching

Git uses branching as it’s version control system. A main branch (previously and sometimes still referred to as master) serves as the known-good, stable, current, source of truth, intent; the master copy of a key for example.

A branch, another full copy of the code with a different identifier from main/master, can be created for development purposes. Bug fixes, feature releases, scaling, or routine changes can be done within a branch, protecting the main branch, and once tested and QA has been performed, the branch is merged through a mechanism known as a pull request, back into main, updating main’s artifacts accordingly.

It might not seem obvious at first but in larger distributed development environments the pull request system allows for cross-team orchestration and collaboration. Pull requests can require approvals and reviews and can then also be used to trigger software builds and releases. The ever evolving history of a piece of infrastructure is completely documented and tracked in the pull request history handling the entire lifecycle of any given product, platform, or host.

My favourite Git-related sites:

Learn Git Branching

Oh Shit Git!

IDE – VS Code

I love VS Code. I really do. After Git is installed download and install VS Code. VS Code is where you will be writing all of your code and reviewing the artifacts your code generates. It is fully Git integrated and aware and things like Git clone, Git add, Git commit, and Git push are all simply point-and-click operations. Split-screen editing, syntax checking, and a vast library of extensions make VS Code my number one pick for an IDE.

VS Code Extensions

Extensions are plug-ins you can install to further enhance VS Code’s already awesome capabilities. There are thousands of extensions available out there. Here is my VS Code extension list that I find helps enhance my infrastructure as code development experience.

My favourite extension:

GitLens

Development Services – Microsoft AzureDevOps

Formerly Microsoft Team Foundation Server (TFS), AzureDevOps provides development services for distributed development teams in the form of work boards (KANBAN; other Agile systems; waterfall), Git repositories, software builds, tests, and software releases allowing for full CI/CD DevOps.

ADO has the advantage, for me, as being an on-prem / private cloud solution with full enterprise controls (RBAC; AD integration) and feature sets.

Git repositories can transition to SSH key authentication. Docker container images can be build and deployed based on Git triggers and actions which build and deploy Ansible playbooks automatically.

Rich history, version and source controls, and an amazing collaboration space – particularly around Git Pull Requests – which fully enable and charge up infrastructure as code development. Adapt and adopt Agile practices to infrastructure teams.

Docker Integration

Moving towards infrastructure as code and full CI/CD in AzureDevOps Docker has become a very important tool and a key component of my success in DevOps. A Docker container image can be thought of as an immutable CD-ROM/DVD-like ISO (hence the “image” part) which can run, self-contained and without the need for a full blown hypervisor like VMWare or Hyper-V, an operating system and software inside of it. Docker images can be interactive and you can “log into” / shell into them, but any changes made inside this session are discarded when the session ends. Ansible and pyATS can both be “containerized” and run inside Docker container images.

Why is this important?

It allows me to setup a software build (create a Docker container image based on a specified Dockerfile) and software release in AzureDevOps CI/CD pipeline. Now any Ansible playbook or pyATS test I previously scheduled with a human operator executing the automation to fully automated and human-independent CI/CD that is trigger based on Git actions like Pull Requests.

A quick approach to Docker:

– Make it work at the CLI.
– Wrap this / convert this to Ansible playbook.
– Wrap the Ansible playbook in Docker.
– Build and release Docker image based on Git actions that trigger the CI/CD.

A sample infrastructure as code build:

And the matching release:

With detailed logs showing the Ansible playbook and Docker status.

Automation Tool – Ansible

After discovering Ansible in April of 2017 my entire approach to solving infrastructure problems changed. Now I work with an automate-first principal and nearly every solution I’ve developed in the past three years has been an Ansible playbook of some kind. It really has been a one-size-fits-all tool for me. Cisco, Microsoft, Linux, VMWare, Azure, anything with a RESTful API; Ansible has done it all for me.

My key points about Ansible:

– Simple, powerful, agentless.
– No previous coding skills required. This is not like learning Python from scratch.
– Can be used for anything from gathering facts, tactical one-time changes at scale, or full configuration management.

My full AnsibleFest 2020 deck if you want to dive deeper.

Ansible-related file types – YAML, JSON, Jinja2

The loaded term “infrastructure as code” or even “network automation” really boils down to the fact that you will be working with a few new file types to create artifacts like data models, templates and playbooks.

YAML Ain’t Markup Language (YAML)

YAML is a human readable data serialization language. In terms of Ansible both your data models (infrastructure represented as intent-based code) and playbooks (the file containing the serially executed tasks to perform) will be YAML files.

A data model for a switch might look like this:

As you can see the file format is simple, made up of lists of key-value pairs, and very human readable. This represents the intent for this particular device.

A playbook, on the other hand, might look like this:

This playbook is scoped for the CampusAccess group in the Ansible hosts inventory file. Prompts for username and password and then runs the ios_facts module printing the gathered facts on the screen.

JavaScript Object Notation (JSON)

You may not need to write JSON but you should be able to consume JSON if you are working with Ansible. All Ansible facts, for example, are represented in JSON. This is much like a RESTful API that returns JSON in response to an HTTP GET. You may need to write JSON if you are POST / PUT (creating / updating) records with an API as the body of the HTTP POST / PUT will be JSON data.

Ansible facts get returned by default like this as an example of an Ansible-related JSON artifact:

Jinja2

Jinja2 is Ansible’s (and Python’s) templating language. Saved as .j2 files, a Jinja template is exactly that – a template of another file type. Configuration stanzas, JSON, YAML, CSV, markdown, HTML, plain-text; more or less anything can be templated with Jinja2.

Logic is often applied inside a Jinja2 template such as For Loops or If Else End If declarations. Jinja2 also allows for the use of variables – which reference the original data model examples.

The VLANs from the data model of example could be templated for Cisco IOS as follows:

{% for vlan in host_vlans %}
vlan {{ vlan }}
name {{ host_vlans[vlan].name }}
{% endfor %}

I have written many more Cisco IOS Jinja2 templates you can check out.

Automated Documentation with Ansible filters – RAW JSON, Nice JSON, Nice YAML, CSV, Markdown, and Interactive HTML Mind Maps

Probably my favourite, and often overlooked, Ansible capability is to generate automated network and infrastructure state documentation. I do this with Ansible filters. Starting with this simple playbook:

This image has an empty alt attribute; its file name is image-17.png

The Ansible magic variable – ansible_facts can be transformed. To simply take the RAW JSON inside ansible_facts you can use the Ansible copy module to copy them into a file:

But using Ansible filters – adding | and then the filter, the ugly RAW JSON can be transformed into “nice” human readable JSON:

Or even better – Nice YAML!

Which looks like this:

CSV and Markdown files can also be created using the ansible_facts JSON and another filter, JSON_Query, an SQL-like query tool used against JSON.

So we set_facts (create our own variables) from the parsed JSON:

Which we can then use to make CSV:

Or Markdown:

Mark Map

Mark Map is a nifty little program that converts any valid, well-formed Markdown file into an interactive HTML “mind map”.

You need to have node.js and npm installed in the Linux environment.

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash –

Then simply run the one line command referencing the markdown file you want to convert to a mind map.

npx markmap-cli <filename>

The output, which is fully interactive, looks like this:

Ansible Vault

A big part of moving to CI/CD and automated builds and releases from human-driven automation is securing the credentials used to authenticate against any given infrastructure. As all of the infrastructure as code is in a central Git repository you don’t want to store your credentials in plain / clear text. One approach is to use prompted mechanisms for securely handling credentials but this does not lend itself to full autonomous automation in Docker containers.

Ansible Vault provides a way to encrypt full files or in our case specific variables, such as the credentials key-value. Once vaulted the encrypted variable can then be safely stored inside the Ansible group_vars file inside the Git repo for all to see. The matching password to unlock the variable can be provided at runtime (again counter intuitive) or, and this is my approach, saved in plaintext in a file in a secure location on the Linux host.

The magic happens at Docker container image runtime where the password file is mounted as a volume into the Docker image so the Ansible playbook can dynamically unlock the credential variables at runtime. Because the lock and key are separate this is a very secure way to automate Ansible with Docker safely.

To move from something like this, that uses prompted inputs from a human operator

Vaulted variables that can be run non-interactively

Use the Ansible Vault mechanism.

$ ansible-vault encrypt_string ‘string’ –name ‘variable_name’

Where ‘string’ is your enable secret for a Cisco IOS for example.

$ ansible-vault encrypt_string ‘My$ecretKey1’ –name ‘ansible_ssh_pass’
New Vault password: <A Strong Encryption password here>
Confirm New Vault password: <Repeat Strong Encryption password here>

Now you can replace the ansible_ssh_pass variable with the vaulted password.

To make it fully non-interactive save your Vault password ( <A Strong Encryption password here>) to a plain-text file (yes it seems counter intuitive but this is fine and safe) saved somewhere secure on the Linux host.

sudo vi /etc/secrets/AnsibleVault.txt
< A Strong Encryption password here >

(ESC, wq! )

Then, in your ansible.cfg file stored in the same location as the playbooks add the following line under [defaults]

[defaults]
vault_password_file: /etc/secrets/AnsibleVault.txt

The playbook will now securely and automatically authenticate without the need for prompts or for insecurely saving naked credentials in the clear.

Latest tool – Application Programmable Interfaces (APIs) for Infrastructure

APIs have finally arrived for infrastructure in the form of RESTful interfaces on switches and routers, on the Cisco Catalyst 9000 series for example, and other appliances and web services. I have had great success with F5’s AS3 API. Cisco.com has amazing APIs. Cisco Prime Infrastructure and Cisco Identity Services Engine APIs are extremely capable and easy to use. BlueCat Address Manager has an API. They are popping up everywhere!

Command Line Interface: cURL

Client Uniform Resource Locator (cURL) is a command-line tool used to interact with an API. As of May 2018 cURL is even included in Windows by default.

Try it yourself – launch a command prompt (Start -> Run; cmd) and type in:

curl https://quotes.rest/qod

You should get back a Quote of the Day in JSON from the public open RESTful API:

Graphical User Interface: Postman

Postman is a GUI API client. Postman can be used for quick and simply API interactions but is also a very powerful API automation and development tool that should not be dismissed as just a simple API browser.

The same Quote of the API would look like this in Postman:

Automation: Ansible URI Module

Ansible has a universal module, the URI module, that allows for API automation. The follow Ansible playbook, quoteoftheday.yml, can be created to automate the Quote of the Day.

Using additional Ansible tools the JSON can be manipulated to create usable reports from the JSON.

The Quote of the Day playbook is available on Automate Your Network’s GitHub.

Quote of the Day API
https://github.com/automateyournetwork/QotD
0 forks.
1 stars.
0 open issues.
Recent commits:

State Validation: Cisco Testing Automation Solution (CTAS)

The Cisco Testing Automation Solution (CTAS) is actually three distinct tool kits assembled in a pyramid / hierarchy.

Siming Yuan blog 3

The foundation here is parsing. Using the Genie library framework various infrastructure CLI commands are parsed and transformed into JSON output. From there pyATS can run automated boolean tests against the Genie parsed returned key-value pairs. xPresso is a GUI based ecosystem that provides for RBAC, scheduling, and much more advanced and easy to build testing workflows.

Similar to Ansible which has connection strings inside group variables CTAS uses testbed files which describe and provide shell connectivity to run the parsing and testing.

A sample testbed file for a Cisco ISR. Note the password is encrypted using pyATS cryptography methods.

A sample crc_errors pyATS test file, written in Python. This test could be used with the ISR testbed to check for CRC errors on all interfaces.

A sample log, taken from an AzureDevOps CI/CD release inside a Docker container image:

xPresso also offers the automation and orchestration capabilities:

Summary

Over the past three years my toolkit as a network engineer has grown dramatically from a humble text editor and a file transfer program to dozens of new tools each with their own amazing capabilities. In short – go get these tools and start using them to solve problems.

– Git
– VS Code
– VS Code Extensions
– GitHub account and repositories for public use
– AzureDevOps for Enterprise use
– Linux of any flavour
– Ansible
– Python
– Docker
– Postman
– cURL
– Ansible URI module
– YAML experience
– JSON experience
– Jinja2 templating experience
– Markdown experience
– HTML experience
– Mind map transformations
– Genie parsers
– pyATS
– xPresso

I am always open to questions, comments, or feedback if you need help getting started!

Downloading the tools and exploring yourself is the best way to get started but I’m here to help!