Category: pyATS

Posted on

My First Pure-Python Network Automation with pyATS / Genie !

I am very proud of this next piece of infrastructure as code for a few reasons.

  1. It addresses the problems with performance and speed at scale I’ve had with my current methodology and tools (Ansible)
  2. I feel like I am ready to “graduate” from Ansible to Python
  3. I’m already using Genie
  4. I’m already using pyATS
    * Limited to the handful of Solution Examples
  5. I already have working automation solutions and I think I can translate / refactor / at least be inspired by previous Ansible-based solutions.

Where to start?

I’ve been down the road of learning network automation from scratch – this time let’s start with simple information gathering and transformation.

Speaking of inspiration – I am going to start with a “Just the Facts” approach and go get – show interfaces status – my favourite command – into a CSV, MD, and this time let’s spice it up and also throw in an HTML page. From Genie parsed JSON.

Only this time using pure Python – no Ansible training wheels (crutches ?)

How to attack this ?

Break it down in human language and then see if we can translate it to Python is one approach. Another is to find working examples and guides provided by the Cisco team. Using a mix of the two and some other online resources here is how I did it.

The job folder is where I will keep the pyATS job file and and code file. Output will hold the 3 output files. I plan on hopefully using Jinja2 just like in Ansible so we need a Templates folder. Finally pyATS uses the concept of testbed files to setup connectivity and authentication. These are very similar to Ansible group_vars.

I’ve included a .gitignore file to keep the .pyc files out of the Git repository.

The Job file. This is a pyATS control file you can use to run the code. You can feed arguments in this way but I have not done that here.

The job file

Pretty simple so far – import the os and run the code.

First thing in the Python code is to setup the Python environment you need. Make sure to import JSON as we need to work with the Genie parsed data.

The actual code

Next we will setup Jinja2 and the File loader

Jinja2 setup

Now we import Genie and pyATS

Setup a logger

Ok so we need 3 source templates one for each file type

Turn on the logger

Let’s load up the testbed file

A testbed looks like this:

Note that yes! We CAN encrypt the string! %ENC{ } represents the pyATS encrypted string! Safe to store in Git repos!

Now some magic – we parse our command into a variable as JSON

Run the results thru the templates

While look like this:

CSV
Markdown
HTML

Then we create the output files back in Python to finish the playbook

Which look like – ha! – we dont know if this works yet! Lets check it out!

The job in action

The command to run the job

Next it loads up the testbed

pyATS is very verbose but in a good verbose with valuable information about your job

Next the actual SSH connection sets up using Unicron (this is different than Ansible which uses paramiko)

Ok my device’s banner is displayed. My banner is left over from some CI/CD work but it’s the right banner – I’m in !

Some basic platform stuff gets dumped to the job log followed by my next job steps

It seems to be working so far
show interfaces status

Ok it’s fired the command! Milestone in the job reached – now it should register this result as JSON in a variable next.

Now during my development I added the following to confirm this step was working to dump the variable to the screen:

print {{ variable name }}

Print replaces debug: msg=”{{ }}” – good!

Similar to an Ansible recap we get a pyATS Easypy Report

Easypy Report > Ansible.log

The Git Add * test

I like to build suspense so I change directories up a folder and try to stage, hopefully, the 3 new files into Git

cd ..

Git add *

Git commit -am “did my first python code work?”

Image

Amazing – but what do they look like?!?

They look incredible!

Image
CSV output
Image
Markdown Output
Image
HTML Output – RAW
Image
HTML Rendered

What does this mean ?

It means, seemingly, I’ve been mastering the wrong tool. That a faster, easier, and more elegant tool is available. This is ok – I feel like Ansible was primary school and I’m moving into the next stage of my life as a developer and moving up into high school with Python.

It also means I have a lot of code to refactor into Python – also fine – a good opportunity to teach my colleagues.

I also means I will be focusing less and less on Ansible I think and more and more on Python

20 years ago I was studying to become a computer programmer analyst in college writing C++, Java, Visual Basic 6, COBOL, CICS, JCL, HTML, CSS, SQL, and JavaScript and now, two decades later, I still have the magic touch and have figured out Python.

You can expect a lot more solutions like this – in fact I am going to see if I can work in my #chatbot / #voicebot capabilities into Python.

Dark Mode

Modern_Show_Interfaces_Status (this link opens in a new window) by automateyournetwork (this link opens in a new window)

A modern approach to the Cisco IOS-XE show interfaces status command using Python pyATS / Genie and Jinja2 templating to create business-ready CSV, Markdown, and HTML files

1 Subscriber 1 Watcher 0 Forks Check out this repository on GitHub.com (this link opens in a new window)

Posted on

Creating Your Own Network Automation ChatBot with Discord, Ansible, and Genie/pyATS!

As you may know I love to play with new toys. I especially love connecting new toys with my old toys. What you may not know is that I am also an avid World of Warcraft fan and player! In order to run what are known as “raids”, group content designed for 10 – 30 players, I use a program called Discord.

My goal was simple – could I send myself messages in Discord from my Ansible playbooks with network state data? Could I create a #chatbot in this way ?

As it turns out not only could I achieve this – it is actually pretty straight forward and simple to do!

Setup

There are not a lot of steps in the setup.

  • Download and install Discord
  • Setup an account
  • Create a Server
  • Create a Channel
  • I named my channel AutomateYourNetwork and set it up as private with invite only RBAC to see the channel
  • Once we have a server and channel setup we need to setup the Integrations
  • Now setup a WebHook
  • Select the channel you want your chatbot to send messages to
  • We will need the Webhook URL

Postman Development

As with all new API development I always start in Postman to sort out the authentication, headers, and body to POST against this new Discord Webhook.

First let’s setup a new Postman Collection called Discord; add a new POST request to this collection

For the request itself make sure you change the default GET to a POST and then use the following URL:

https://discord./com/api/webhooks/< your URL copied from Discord here>

The body is flexible but to get started let’s just use a small set of values.

Set your body to RAW JSON

And add this body (change your username unless you want this message to look like it came from me!!)

Now if you really want to see how fast / real-time / amazingly cool this is – make sure you have your Discord logged in but minimized to your system tray

Hit SEND in Postman

Your Discord should have notified you about a new message! In my system tray the icon has also changed!

Which is no surprise because back in Postman we see we received a valid 204 No Content response from the API

Lets see what Discord looks like

How cool is this?!?

Integrating with Network Automation and Infrastructure as Code

Ok this is great – but can we now integrate this into our CI/CD pipeline Ansible playbooks?

Can I send myself network state data ? Can we create a #chatops bot ?

Yes we can!

Lets start with Ansible Facts – and see if we can chat ourselves the current version of a device.

First let’s setup our credential prompts

Then, let’s use ios_facts

Thats more or less all I need to do – next let’s use the URI module to send ourselves a chat!

I will break down this next URI task; first setup the URL – again after /webhooks/ {{your URL here }}

This is a POST

I like to do this next step for two reasons; one to set the body of the POST to JSON (obvious) but also to allow me to use YAML syntax in Ansible to write the body of the POST (not so obvious). Without this my body would need the JSON formatting (think moustaches and brackets) which is hard enough to write on it’s own, and very hard to write inside a YAML Ansible task

Meaning I can format the body as such (in YAML):

And, like we saw in Postman, we are expecting a 204 back and no content

Make sure you are delegating to the localhost (you dont want this step to run on your Cisco switch)

Again back to the body we are accessing the Ansible magic variable ansible_facts and the key net_version

Lets run the playbook!

Discord looks happy

And, with the power of Internet Magic, here is our message!

This is incredible – let me integrate Cisco Genie / pyATS now and send some parsed network state data next – to show the real power here

The playbook structure is more or less the same so save the Ansible Facts version playbook and copy / rename it to Show Int Status.

Keep the prompts; remove the ios_facts task and replace it with this task

Followed by the Genie parsing step

Then we need to adjust the Discord message – for my example I only want a message if, for example, an interface is configured to be UP / UP (meaning it is not administratively down) but is DOWN / DOWN (notconnected state). I don’t care about UP/UP or Administratively Down interfaces.

Again, I will break down this

Most of this is the same

Here comes my magic with Genie. We want to loop over each interface Genie has parsed into the registered variable pyatsint_status_raw.interfaces. We need to convert this dictionary into a list of items so filter it | dict2items

Now we want a condition on this loop; only call the Discord API when the {{ item.value.status }} key (that is to say each iteration in the loops status value) when it equals “notconnect“

Now we can reference the item.value for the per-interface name and the item.value.status for the notconnect status when it hits a match in the body of the message we are sending to Discord.

The task as a whole looks like this:

So we run this new playbook which sort of looks like this. Remember we have a condition so the light blue text indicates non-matches / skipped interfaces (because they are connected or admin down); green indicates a hit.

Drumroll please

And now in Discord I have this wonderful, pager-like, real-time “alert”

Now go build one!

Here is the GitHub repository – go try to build one for your network!

Dark Mode

DiscordNetworkChatBot (this link opens in a new window) by automateyournetwork (this link opens in a new window)

Ansible playbooks that chat with Discord using Ansible, Genie/pyATS, and the Discord webhooks that send network state information as a Discord message!

1 Subscriber 1 Watcher 0 Forks Check out this repository on GitHub.com (this link opens in a new window)

I would *love* to see your #chatbot in action – please hit me up on Twitter and show me what you got !

Posted on

Collaboration is the key to automation success – A Genie Success Story

I’ve been on about a three year journey with network automation and while I have had great personal and technical success – my organization and most of those outside my immediate day-to-day circle are still shackled to the ‘old’ way of doing things (primarily the CLI).

This year I decided to start a new program – a training session for those outside of my small development team – primarily targeted at the “operations” staff who can benefit the most from automation and infrastructure as code. This includes network operators, monitoring / NOC team members, IT Security staff, other developers, compute (server / storage) teams; everybody is welcome. We divided the calendar up into different teams with different recurring timeslots.

In advance I had written and tested a bunch of Code for the Campus Core – Ansible Facts and Genie parsed show commands – transforming the output into business-ready documentation. My plan was simple enough:

1. Ensure everybody was on the same page and had the same toolkit
* VS Code
* Git
* Azure DevOps repository bookmarked
* Various VS Code extensions
* A basic overview of main vs working branches in Git
* A basic outline on Ansible, YAML, Jinja2, and JSON

2. An operator would create a working development branch – in our case the Distribution Layer – so dist_facts Git branch.
* This operator ‘drives’ the whole session sharing their screen
* Step by step, line by line, refactoring (fancy way of saying copy-pasting) working code from the Core and updating it for the Distribution Layer as necessary
* Git clone, add, commit, push, and pull performed in both VS Code and Linux CLI
* Ansible playbook executed with a –limit against one building, then at scale after validating output
* Thorough tour of the JSON, YAML, CSV, MD, and HTML files after each run

3. Work through Ansible Facts and various Genie parsed commands to build up a source of truth

So far it has been a very successful approach with the two teams adopting Marvel superhero teams (TEAM: CAPTAIN AMERICA and TEAM: IRON MAN respectively) allowing me to create memes like this:

Image

Anyway – back to the point – today we had the following exchange:

Me: “So – we just parsed the show etherchannel summary CLI command and transformed the output into CSV files – amazing right! Any questions?”

Operator catching on quickly: “We use the show interfaces trunk command often to track down what VLANs are being carried on which interfaces – can we transform that into a CSV?”

Me, excited and proud of the Operator: “Amazing question and I’m glad you brought up a practical example of a command you use in the field all the time we can maybe transform into something a little more useful than CLI output!

Launch the Genie parser search engine (under available parsers on the left menu) and let’s see if there is a parser available

Bingo! Let’s do it!”

The playbook

In this example we are targeting the Campus Core.

The playbook is simple enough

Use the Ansible ios_command module to issue and register the show interfaces trunk command

Next, using the Genie filter plugin

Filter the raw variable and register a new variable with the Genie parsed results

Note you need to pass the parse_genie filter two arguments the command itself and the appropriate Cisco operating system

Next I like to create a Nice JSON and Nice YAML file with the parsed results as follows:

Which look like this:

And this:

I then use a loop to create a CSV and MD file from a Jinja2 template

The Jinja2 Template looks like this

Couple things to hightlight:

* We challenge the current iteration of the Ansible loop and when it is on “csv” we template a CSV file format otherwise (else) it will be “md” and we template a markdown file
* The For Loop is over each interface in the results
* Be careful with the CSV file you need to regex_replace the comma out of results because they are comma-separated which will throw off your CSV file. The markdown does not require any regex.

Which results in this amazing sortable, searchable, filterable, version controller, source controller, truthful, fact-based CSV file:

Now this example is just the singular logical Core but we will quickly refactor the code next week in our next session together and the operator who wanted this playbook will get a chance to write it ! Then we will have the interface trunk information for the entire campus automatically in spreadsheets!

The moral of this story is to collaborate. Ask your front-line operators how automation can help them. Do they have any frequently used or highly valuable CLI Commands they want transformed into CSV format? Would Ansible facts help them? And then show them how to do it so they can start writing these playbooks for themselves.

Posted on

Excitement

And he’s right! I can’t stop making new GitHub repositories with Genie parsed show commands to documentation! Like show ip interface brief as seen above!

An Ansible playbook that uses Genie parser to convert Cisco IOS show ip interface brief command to documentation
https://github.com/automateyournetwork/IOS_GenieParsed_Show_IP_Int_Brief
0 forks.
2 stars.
0 open issues.

Recent commits:

Posted on

Letting the Genie out of the bottle

Imagine if you could transform that unstructured Cisco show command screen output into something more useful than just CLI output.

What if there was a way to transform an SSH CLI show command’s response into a RESTful API-like HTTP GET request response in RESTful API-like structured JSON?

Sounds amazing right? Well with Genie this is totally possible. I mentioned the CTAS Genie / pyATS / xPresso solution in My Toolkit post. I also suggested that gathering facts with Ansible is the best place to start with network automation and infrastructure as code.

But the Ansible facts, while impressive, rich, plentiful, and extremely useful, they do not necessarily contain all of the state information that IOS / NXOS CLI show commands provide. Some information, such as CDP neighbors, interfaces, IP addresses, is available with only the ios_facts / nxos_facts modules but for things like the configured Virtual Route Forwarders (vrf) on a router, the IP Address Resolution Protocol (ARP) tables, or the OSPF routing tables you are stuck with crappy old Cisco CLI output right?

Wrong. You now have a magical Genie on your side who is here to grant all your state capture and transformation wishes! And you get more than 3!

TL;DR

– The historic restrictions of using Cisco IOS / NXOS show commands as CLI-only, raw screen / putty logged output, have been lifted.
– Genie parsers provide REST API HTTP GET-like responses to common CLI show commands.
– Ansible integrated allowing for running and parsing show commands at scale.
– I like to create RAW JSON, Nice JSON, Nice YAML, CSV, Markdown, and interactive HTML mind maps from any JSON I can get my hands on. Now I can do it with Cisco show commands!
– Fill the gaps from what is missing from base Ansible facts.
– Build a powerful, dynamic, state aware documentation set for every device in your network from every day show commands.
– Not as difficult as you might think.
– Another modern network automation, infrastructure as code, tool network engineers should include in their skillset.
– The best development team in the business. The Genie / pyATS / xPresso developers have personally helped me out. Find them on Cisco WebEx Teams for often real-time responses.

What is Genie?

Genie is a parser that automatically converts Cisco IOS / NXOS command output into structured JSON. This structured JSON allows developers to then work more programmatically with the output from important, but otherwise useless, show command output.

For example I am using Genie to parse some key show commands and create a dynamic automated library of documentation in different formats.

You can also go a step further with pyATS and write boolean tests (true / false) in Python using the Genie parsed data as your source of test data. The show ip ospf neighbor command, for example, can be executed, parsed with Genie, and then tested with pyATS! All of this can then be wrapped in business logic, scheduling, and protected with RBAC in xPresso.

Amazing but I am not made of money – how much does all this capability cost?

It is all free.

How do I integrate it with Ansible?

The amazing Ansible integration that I am using is thanks to Clay Curtis and his incredible contributions. Thanks to Clay there are two installation steps on top of the standard Ansible installation and an open Python filter plugin – then you are ready to go.

Please visit the Ansible Galaxy role, Cisco DevNet Code Exchange, and GitHub repository for all the details.

Show and Tell

It’s easier to just demonstrate how the Parser can be used with Ansible. Some prerequisites:

– Linux host
– pip install ansible
– pip install genie
– ansible-galaxy install clay584.parse_genie
– SSH access to network devices from this host
– Credentials for the device (prompted)
– The parse_genie Python filter_plugin
– Make sure your ansible.cfg specifies the location of the parse_genie file in filter_plugins.

[defaults]
filter_plugins=../filter_plugins

Example: show vrf

Recall what a show vrf looks like at the CLI:

This could spawn for pages depending on how many VRFs are hosted on the router. Also – how do you work with this output? Log to a putty file and inspect in notepad offline? Not very user friendly.

Let’s Genie parse that same command and see what the output looks like as structured JSON and take the Pepsi Challenge against the CLI.

In a playbook called CiscoVRFFacts.yml I first scope the playbook (CampusDistribution), then prompt for username and password. Note the collection includes Clay’s clay584.genie collection.

Next I run my standard Cisco show command with the ios_command module and register the response (which is RAW unparsed IOS config at this point) Nothing fancy here.

The next step is where we use the filter_plug in to parse the registered raw response and register a new variable that holds the parsed output. Again – this is not very complicated once you understand the syntax.

Note the parsed command is the same as the original command, in this case show vrf, and we have to specify the operating system (Cisco IOS).

You can optionally print these parsed facts, the nice JSON, to the screen.

Resulting in something like this:

We can save this output to a variety of files and manipulate the output in a variety of ways previously unavailable to us with raw standard CLI IOS output.

For starters lets put the RAW JSON response into a JSON file with the Ansible copy module.

Which is good for forensics, audits, security purposes, or for downstream systems that intake raw JSON, but it’s not very human readable.

Add the Ansible filter | to_nice_json to get the output to look like the output that is printed to the screen.

Now this is up for debate but personally I prefer and find YAML even more human-readable than JSON. Let’s make a YAML file with the | to_nice_yaml filter.

As a reminder this is what the show vrf command looks like at the CLI:

This image has an empty alt attribute; its file name is image-53.png

Now, in YAML:

Incredible!

Going a step further we can try to manipulate the output for business suitable reports in CSV, markdown, and HTML files.

Using yet another Ansible filter, the dict2items, which as the name implies transforms a dictionary to a list of items, we can loop over the original variable {{ pyats_all_vrfs.vrfs }} key and create our CSV / markdown.

(There are some Regular Expression (RegEx) steps that clean up the JSON a bit omitted for brevity)

Add a header row.

And now you have a CSV file!

Similar steps can create a markdown.

And then an HTML mind map can be generated.

Look at all the business and operational value we’ve squeezed out of a simple show vrf command!

All of this code is available on Automate Your Network’s GitHub.

An Ansible playbook that uses the Genie parser to transform the Cisco IOS "show vrf" command into human readable documentation
https://github.com/automateyournetwork/IOS_GenieParsed_Show_VRF
0 forks.
1 stars.
0 open issues.

Recent commits:

Example: show ip arp

Start with the CLI show ip arp command output, which to be fair isn’t the worst CLI output around, which provides the ARP table for the Global Routing Table.

With more or less the same steps transform this into the same reports.

Setup the playbook:

Run the show ip arp command:

Parse it:

Create your 3 base RAW JSON / Nice JSON / Nice YAML files:

Check out this nice output!

Now anybody, CCNA level or not, can read the ordered structured list and see that VLAN20 has 1 neighbor with an IP of 172.24.2.1, the age, and the MAC address.

Similar steps to transform the output create the CSV / markdown / mind maps:

Also available on GitHub.

An Ansible playbook that uses the Genie parser to create reports from the show ip arp Cisco IOS command
https://github.com/automateyournetwork/IOS_GenieParsed_Show_IP_ARP
1 forks.
1 stars.
0 open issues.

Recent commits:

Example: show ip arp vrf {{ vrf }}

The exact same steps can be performs by simply adding show ip arp vrf <vrf name> with the same output as the Global Routing Table.

As a development aside I had big plans for show ip arp vrf {{ vrf }} to be a dynamic and automatically loop over all of the VRFs present on the route. I got pretty far but the parser itself hung me up.

Meaning I had a previous loop over the Genie parsed show vrf command which provided me the VRF name to feed the show ip arp vrf command. This all worked out and I could get the raw unparsed list like this:

ios_command:
commands:
– show ip arp vrf “{{ item.key }}”
loop: “{{ pyats_all_vrfs.vrf | dict2items }}”

But when it came time to parse this the following didn’t work.

| parse_genie(command=’show ip arp vrf {{ item.key }}’, os=’ios’)

I think because the parser is treating {{ item.key }} as raw text / the raw command and is not aware of the outer loop and to treat it like a variable. For the same reason I couldn’t write it to prompt for a VRF name. So, yes, I found one edge case drawback where I have to hardcode the VRF. Leave me a note if you see a way around this.

Summary

Genie parsers allow network engineers and operators to transform simple Cisco show commands into REST API-like requests with corresponding structured JSON response from the switch. This all magically happens behind the scenes allowing developers to actually make use of the show command output.

The days of setting up your Putty session to log your console to text files for offline analysis of raw standard output are finally over. Now you can work with JSON which in turn can be transformed into YAML, CSV, markdown, and HTML easily.

Along with Ansible facts, Genie parsed state data can solve the lifelong challenge of creating and maintaining good documentation.

Bonus code – show etherchannel summary

I couldn’t help myself and wrote another GitHub repository after finishing the blog. That’s how much I love Genie and how quick and easy it is!

An Ansible playbook that uses the Genie parser to transform Cisco IOS show etherchannel summary into usable reports.
https://github.com/automateyournetwork/IOS_GenieParsed_Show_Etherchannel_Summary
0 forks.
2 stars.
0 open issues.

Recent commits: