Creating a Network Search Engine

Imagine being able to use a keyword search engine against your network ? A Google-like query for “VLAN 100”, a MAC address, IP address, even an ACL or simply the keyword “Down” that returns real time search results !

It sounds far-fetched but that is exactly what I’ve been able to do in the latest addition to my open source project Merlin ! I’ve made this available as open source!

Dark Mode

merlin (this link opens in a new window) by automateyournetwork (this link opens in a new window)

Network Magic: Transforming the CLI and REST API using Infrastructure As Code automation

High Level Goals

  1. Use the pyATS framework and Genie SDK to:
    a. Learn features
    b. Parse show commands
  2. With the JavaScript Object Notation (JSON) we get back from pyATS
    a. Index the JSON into a database
    b. Index the JSON into a search engine
    c. Visualize the database

Enter: Elastic

As you may know Merlin already creates a no-SQL document database using TinyDB – a serverless database that is very easy to use. My only problem is that I haven’t found (and confirmed by TinyDB author) a UI or frontend to consume and present the TinyDB.

Poking around the Internet I found Elastic – a suite of tools that seem like a perfect fit for my goals. “Elastic – Free and Open Search”

I suggest you start here and read about the ELK Stack

The Solution – Elastic Setup:

I setup a 14-day trial in the Elastic Cloud for the purposes to getting going. Elastic can also be run in a local Docker container or hosted on Linux.

  • Note – I tried using WSL Ubuntu but systemd is not currently supported and you will get this error:
System has not been booted with systemd as init system

Once you have logged into Elastic, you can use a Google account for this, you will want to setup a Deployment

Here is Merlin as a Deployment

Which then opens up a full menu of amazing features and capabilities

Some key information:

When you first setup your Deployment you will get one-time displayed credentials you *need* make sure you capture this information!

Your Endpoint (the URL to your database) is available in a click copy here in the main dashboard. You can also launch Kibana and Enterprise Search / copy their unique endpoint URLs here.

Since we are using Elastic Cloud make note of the Cloud ID

As we need this in Python to connect to our endpoints.

In order to setup the Search Engine click Enterprise Search and then when presented with the option Elastic App Search

Create an Engine

Name your engine (I would suggest whatever you named your deployment -engine or -search)

Now the next screen will present you with four methods of populating the Search Engine with JSON

We are going to be Indexing by API and if you pay attention to the Example it will give you what you need to do this and a sample body of JSON

(You get your URL and Bearer Token; make note of both we need them in the Python)

The Solution – The Python:

Here is the relevant Python / pyATS code you need to build your own Elastic index (Deployment) and then also the ElasticSearch search engine !

First you need to pip install pyATS, Elasticsearch, and elastic_enterprise_search

pip install pyATS[full]
pip install elasticsearch
pip install elastic_enterprise_search

Next, in the actual Python, you will need to import the above libraries into Python

As well as the pyATS framework

Next in the Python we need to setup a few things to interact with Elastic

Now we get into the actual pyATS job first setting up the AE Test section and using testbed.connect to establish our SSH connection to the network device

Next we setup our Test Case and define self, testbed, section, and steps. Each Step is a boolean test in pyATS.

For device in testbed kicks off the loop that runs the commands per device in the testbed topology (list of devices in the testbed file)

Now I have defined a function that can be reused that has a step and try to device.learn(function_name).info (and fail gracefully if the function could not be learned)

Now we simply feed this function the various features we want to learn

In this case it was written for the Cisco DevNet Sandbox – NXOS – Nexus 9k which only supports a limit number of features. In a real environment we can learn even more!

Then we use a different function for the parsed show commands

And run a variety of show commands

Now Merlin has, to date, created business-ready documents (CSV, markdown, HTML) and experimental documents (Mind Maps, Network Graphs) from the JSON we have inside all of these variables.

Now here is how we send the JSON to be Indexed in Elastic

Lets take a few examples – learn BGP – as a second fail-safe check in case it did parse correctly but for some reason was empty I first check if its not None

If its not none, we index it in our Deployment

Then we index it in our Search Engine

Here is show ip interface brief

Its easy and repetitive code – so much so that I will likely write another function for these 6 lines of code and just feed it the learn / show command.

The Outcome – Elastic

In order to confirm your Elastic Deployment is up – you can use cURL or Postman or the Elastic API Console

Wait what? I have just built a database that has an API I can query with Postman???

Y E S !

Check this out

Launch Postman and setup a new Collection called Elastic

Add your username and password (the one-time displayed stuff I told you to write down!) under the Authorization – Type – Basic Auth

Add a Request called Deployment

Copy and Paste your Elastic endpoint ID

Paste it in as a GET in your Deployment Requst

You should get a 200 Status back

And something like this

In Elastic – You can do the same thing!

Launch the API Console

If you leave the field empty and click Submit you get the same status data back

What about our Network Data?!

Now if you pay close attention to the pyATS and Python logs – you will see this call and URL (and status) when you send the data to your Deployment to be Indexed

The 200 means it was successful – but you can take this string into Postman / Elastic API Console !

Back in Postman

Which gives us:

And in the API Console we just GET /devnet_sandbox_nexus9k/_doc/show_ip_interface_brief

Now – check this out – make your next GET against just the base index

In the DevNet Sandbox there is almost 35,000 rows of data ! WHAT !?

The full state as JSON

Over in API Console

Very cool what about the Search Engine??

Well the engine is now populated with Documents and Fields

Which look like this

We can filter on say VRF documents and the search engine magic starts

Now lets check out keyword searches in the Query Tester


How about an IP Address

What about “Up”?


I want to be open I have not totally developed out any visualizations but I want to show you Kibana and the absolutely incredible dashboards we can create using the Elastic Deployment data

Launch Kibana and then select Kibana again

Now take a look at the incredible things we can do

As I said I have barely scratched the surface but lets look at what we could do in a Dashboard

First thing we have to do is create an Index Pattern

I’ve selected the devnet_sandbox_nexus9k to be my index pattern

Now I have 6670 fields (!) to work with in Kibana Dashboards

Now it becomes, for a beginner like me, a little overwhelming simply because of the vast choices we have to work with this data


Kibana discovery and learning aside my adventure into network search engines was fun and I learned a lot along the way. I’ve made a video of my development process here if you would like to check it out before you try it yourself. Must Read !

When I was tagged on Twitter about @ioshints (Ivan Pepelnjak (CCIE#1354 Emeritus)) latest blog post I thought somebody was telling me I should read the latest blog

I flagged this as “Hey what a coincidence I was just writing about #chatbots with Discord – I gotta read this later”

Turns out it was my article that was Worth Reading!

Why this is so special to me is that I started my automation journey with an subscription which was a very key part of my early success with Ansible and Cloud automation specifically. Ivan has also personally helped me write better code and taken a personal interest in my success.

I am so incredibly humbled and thankful for Ivan’s recognition but even more by his commitment to be honest and open with his vast knowledge.

In a lot of ways I am trying to emulate Ivan’s approach and appreciate having a virtual mentor of such quality and capability.


Discussing the Future of Network Operations with #init6

I had a chance to meet some of my heroes at #init6 including Daren Fulwell!

Make sure you checkout my open source project fueling the demo

Your Automation Journey can start anywhere!

So you want to automate all the things in your network? Great! more and more enterprises are realizing the benefits of automating the network and are demanding more from their engineering staff. You’re also probably thinking about your career as well and how these automation skills will really come in handy to progress your career. Like I said, Network Automation skills are in high demand now.

So now what?

If you’re like me, you’ll jump straight into it and quickly realize that automating what can be considered the “life and blood” of an enterprise can be a daunting task. Rest assured though, there are things you can do today to begin your automation journey without needing to jump right into the deep end.

Standardize and Document

It would be remiss if I start off by saying that JC in his book “Automate Your Network” recommends starting your Automation Journey off by documenting and standardizing anything and everything in your network. For me, a big benefit of doing this before I dove into python and scripting was that I was able to bring my environment into a standard that makes building automation a very simple task.

I admit that it took me a bit of time and investment to bring my network into a working standard but once things were aligned it made the code I wrote that much more reusable.

Template all the Things!

If your like me, than you probably have a bunch of standard configuration commands stowed away in an Excel Document. You may not know it but one simple thing you can do as your start the automation journey is to convert those Excel Documents into clean (and reusable) code using the JINJA2 python library.

By templating your commands with JINJA2 you’ll immediately see the sky is the limit when it comes to where those commands can be run from. Whether you push those templates to another Python Library like pyATS / netmiko or simply document them for other team members, you’ll quickly see the benefits of “codifying” those CLI commands at an early stage.

Create Tooling for your Team

As a Senior Engineer, whenever I see a team member create a little tool or even a dashboard I get all jittery and excited. In my opinion, enabling your team is a selfless task that is often forgotten about. This is something I recommend you do (regardless of where you are along your automation journey) as it will get you working with other frameworks and you’ll be one step closer to a more automated and programmatic environment.

Reimagining The CLI with Python

The Command Line Interface. A near 60-year old technology a majority of the world’s networking teams solely rely on for their architectures, designs, monitoring, operations, and support. Most of us, pre-DevNet, have been raised and groomed to become CLI wizards and warriors with a super-hero level of ability being the standard expectation.

Command-line interface – Wikipedia

But the dirty secret is the CLI sucks and we are literally using a tool that was released half-a-century-ago. Imagine if COBOL (1959) was still the only programming language in the world ? Welcome to Enterprise IT Networks – here is the CLI – good luck to you!

Well today I am going to reimagine a modern, next-generation, CLI using Python and see if I can breathe new-life into our very old friend the CLI.

This post in Meme format

Why does the CLI still suck?

It is grounded in it’s historical roots. Take for example that the original IOS had 256kB of memory with pre-Pentium chipsets! So they needed a lean and text-driven CLI to configure and operate the early routers and switches.

Cisco IOS – Wikipedia

It is also why the modern next-generation devices come with a REST API that supports RESTCONF, NETCONF, and YANG models – they want to you get away from the CLI and move towards a modern approach.

But what about the legacy fleet? The brownfield? What if my staff are religious about the CLI ? Or fearful to learn REST, start using Postman and JSON, or even dreaded network automation like Ansible or Python ?

Culture will change over time but we need an easy, accessible, and palatable alternate to SSH’ing into devices one by one to perform CLI operations.

Take the most basic, first-to-be-run on any device, command show version.

In this video, the first I’ve ever done, I explore how, using Python, pyATS, Genie, Jinja2 Templates, and Cloud REST APIs, we can create the next-generation show version command – check it out!

BlueCat Cisco Live Contest!

I am sharing this because I love both BlueCat and Cisco!

CiscoLive changed my life and where I was introduced to DevNet and automation!

Make sure you get into this giveaway !

Good luck!

Two more positive reviews!

For whatever reason it seems to be difficult to get even just a “Star” review on Amazon – even harder to get actual comments – but today I received two more 4-5 star reviews!

Thanks again for the support! If you enjoyed the book – or if you didn’t enjoy the book – I would love it if you could take the time to add a star (ok 5 star) review or your feedback on Amazon !

Hello World!

My first Hello World! was not actually a true programming language test – in fact it was the first message I sent over telephone lines on my 2400-baud modem to another BBS (Bulletin Board System for anybody not familiar with pre-Internet computer networking) system admin who was helping me setup “The Backspace BBS” back in 1990. Seeing the words “Hello World!” typed out (slowly) across the monitor of my x286 – typed by “somebody else” out there in the world behind their computer was a memory I will never forget. My fascination and lifelong journey with computer networking had begun!

This Hello World! is equally miraculous given I registered the domain name, hosting space and a Linux platform. From there I installed WordPress on my Linux host out in the cloud somewhere, all via my browser with easy point and click steps. A few minutes of WordPress customization later and boom – I’ve established a place where I can share my journey in this world as a network engineer, and, more specifically, my transformation and metamorphosis into more of a solution developer using DevOps tools and treating infrastructure as code. It is a pretty amazing world we live in where I can do all of this from my home office in about 45 minutes from scratch.

Here I will share my thoughts on infrastructure as code and network automation; discuss the tools and techniques I am using to achieve my goals, and provide as much guidance and insight into my process. In addition to being a Sr. IT Planner and Integrator for the Canadian House of Commons (Parliament) (but this site is strictly my personal thoughts and opinions which are *not* endorsed by the House of Commons), I am also an author. Check out the Books link for information about my first book, “Automate Your Network: Introducing the Modern Approach to Enterprise Network Management”, as well as my upcoming follow-up “Automate Your Network: APIs for Infrastructure”.

Thank you for joining me on this journey – we are going to learn a lot from each other !

I wonder what my next Hello World! will be created with! Only time will tell!