Every network engineer knows the quiet tension of a 2:00 AM maintenance window.

You have reviewed your Method of Procedure (MOP), secured change-advisory board (CAB) approval, and executed configuration updates across core routers. To verify, you run show ip interface brief, confirm all interfaces are up/up, and send a standard 5-packet ICMP ping across the transit link:

Sending 5, 100-byte ICMP Echos to 172.16.30.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5) 

With five clean exclamation marks, everything appears healthy. You close your terminal, update the change ticket to "Completed," and log off.

Three hours later, morning business traffic ramps up, customer service lines light up, and the critical incident bridge convenes: enterprise web dashboards and database sessions between HQ and the Data Center are hanging and timing out.

What went wrong?

The control plane never dropped. OSPF remained in FULL state. BGP remained Established. Small 100-byte monitoring pings passed cleanly. However, an engineer had inadvertently committed ip mtu 1400 along with no ip unreachables on a core transit link while masking the mismatch using ip ospf mtu-ignore.

As soon as real application traffic arrived with 1500-byte frames and the Don't Fragment (DF) bit set, the packets were silently dropped. Because ICMP unreachables were disabled, Path MTU Discovery (PMTUD) failed silently.

Industry studies consistently reveal that over 70% of network outages are self-inflicted during scheduled maintenance windows. The root cause is rarely a complete physical link collapse; rather, it is subtle "gray failures" that easily slip past manual check sheets.

In this article, I walk through how I designed and built netops-ai-mop-auditor, an open-source, zero-trust Pre/Post Maintenance Verification & AI Change Auditor using Python, NAPALM, Netmiko, Flask, and local Llama 3.2 (via Ollama), validated end-to-end inside a GNS3 multi-AS Cisco IOS lab.

For Beginners: The Learning Path from Python Scripts to AIOps

If you are new to network automation—setting up virtual Cisco labs or writing your first Python connection scripts—AIOps becomes intuitive once the underlying mechanics are clear.

This project represents the practical evolution of a foundational multi-part automation series I previously published on LinkedIn. If you are starting from scratch or want to follow the progression step by step, start with these guides:

Step Topic / Focus LinkedIn Walkthrough
1. Lab Virtualization GNS3 + VM Setup & Bridging GNS3 2.1 GUI & VM Installation Steps — Setting up GNS3 and bridging virtual routers to your host machine via Cloud1 and Hub1.
2. Python CLI Basics Raw Socket Connections Network Automation: Telnet to Cisco Router — Understanding programmatic command execution and terminal handling.
3. Secure Automation Enterprise SSH via Netmiko Network Automation: SSH via Netmiko — Managing SSH sessions, handling command prompts, and navigating privilege levels.
4. Multi-Vendor State Structured Config with NAPALM Network Automation: Configure Router with NAPALM — Moving beyond screen-scraping toward structured multi-vendor getters and config management.
5. Web Dashboards Telemetry via Nornir + Flask Network Automation: Telemetry via Nornir & Flask — Surfacing live router metrics inside an interactive browser UI.

Once you understand how to interact programmatically with network devices and render operational state on the web, the next engineering challenge is: How do we introduce Large Language Models (LLMs) into NetOps workflows safely without risking network downtime?

Why Naive "LLM Chatbots" Fail in Production NetOps

When engineers first experiment with AI, the common approach is to copy and paste hundreds of lines of show running-config before and after a change into an LLM with a prompt like:

"Here is the configuration before and after my change. Did anything break?"

In real-world network operations, this naive approach fails due to two distinct engineering hazards:

1. The Hallucination & Arithmetic Hazard

Small local models (such as 3B to 8B parameter models) are probabilistic language models, not deterministic calculation engines. When presented with 600 lines of unstructured Cisco IOS configuration text, they regularly miss subtle wildcard mask typos, miscalculate BGP prefix counts, or hallucinate non-existent interface names. Relying on an LLM to accurately diff routing tables in context memory is an architectural anti-pattern.

2. The Unacceptable Blast Radius of Autonomous Write Access

Granting an AI model autonomous write access (configure terminal) to enterprise core infrastructure is a risk no prudent engineering team will permit. A single prompt injection or hallucinated command syntax could trigger an unrecoverable routing loop or widespread blackout.

The Solution: The Anti-Hallucination Split Architecture

To make AI production-grade in networking, we enforce a strict separation of concerns:

  • Deterministic Python Engine (NAPALM + Netmiko + difflib): Machines excel at structured data collection, cryptographic snapshot hashing, mathematical line-by-line diffing, and synthetic data-plane probes.
  • Local Semantic Auditor (Llama 3.2 via Ollama): LLMs excel at reasoning over structured context, auditing semantic scope compliance against authorized human change requests (MOPs), and writing human-readable Post-Implementation Review (PIR) reports.

System Architecture

The GNS3 Lab Topology: Multi-AS Enterprise Core

The validation environment is hosted in GNS3 and simulates an enterprise-to-data-center transit architecture:

GNS3 Lab Topology

  • R1 (AS 65001 – HQ Edge): Advertises HQ LAN 192.168.10.0/24 into eBGP across Fa0/0 (10.12.0.1/30) to R2.
  • R2 (AS 65002 – Core ASBR): Bridges eBGP (AS 65001) to the internal core. Runs OSPF Area 0 on Fa2/0 (10.23.0.1/30) to R3 and peers via iBGP (Loopback0: 2.2.2.2 to 3.3.3.3) applying next-hop-self.
  • R3 (AS 65002 – DC Edge): Advertises DC Server subnet 172.16.30.0/24 into BGP.
  • Out-of-Band (OOB) Management Plane: Interfaces R1 e2/0 (172.20.40.211), R2 e1/0 (172.20.40.212), and R3 e1/0 (172.20.40.213) connect through Hub1 to host loopback adapter Cloud1 (172.20.40.0/24), completely isolated from data-plane traffic.

Zero-Trust Engineering Controls

To guarantee operational safety, the platform implements three strict design controls:

1. The 4-Layer Snapshot Engine (tools.py)

Every Pre-Check and Post-Check captures four distinct operational layers across R1, R2, and R3:

  • Configuration Layer: Cleaned running-config (volatile Cisco IOS byte-count headers and timestamps are stripped so clean diffs produce zero noise).
  • IGP Layer: OSPF neighbor Router-IDs and FULL adjacency states.
  • BGP Overlay Layer: Peer session states, received prefix counts, and best-path (*>) validity for critical HQ and DC prefixes.
  • Data Plane Layer: Synthetic 100-byte reachability pings and progressive 1500-byte DF-bit Path MTU probes in both directions.

2. Regex-Enforced Read-Only Guardrails (guardrails.py)

The AI agent has zero write tools. Every CLI query requested by Llama 3.2 passes through strict Python validation that permits only show, ping, and traceroute commands while blocking shell escapes (;, | redirect) and write verbs (configure, reload, write, erase) using word-boundary regex patterns.

3. 100% Air-Gapped Local Inference

Running Llama 3.2 locally via Ollama ensures that router configurations, internal IP schemes, and operational telemetry never leave your local management perimeter.

The 4-Stage Maintenance Lifecycle in Action

The Flask web dashboard (featuring Dark/Light mode and real-time Server-Sent Events streaming) guides the engineer through a strict 4-step workflow:

  1. Select Maintenance Scenario (MOP) & Scope: Pick a planned activity (MOP-101 link hygiene, MOP-102 new DC subnet onboarding, MOP-103 iBGP tuning, or MOP-104 custom manual GNS3 work) and define the Authorized Change Scope.
  2. Run Pre-Checks (PRE_ACTIVITY): Lock the 4-layer baseline. Once verified, the UI displays a green CLEARED banner authorizing the engineer to begin work.
  3. Execute Activity (Manual GNS3 CLI or Simulation): Make real changes directly inside GNS3 consoles—or test with the built-in Clean vs. Misconfiguration simulators.
  4. Manual Engineer Confirmation → Post-Check & AI Verdict: When finished, the engineer clicks Confirm Changes Done. Python captures POST_ACTIVITY, builds the Pre vs. Post Comparison Matrix and Unified Config Diff, and streams Llama 3.2's audit verdict.

Case A: Clean Maintenance Approved (MOP-101)

When only the authorized interface description on R2 FastEthernet2/0 is updated, the Comparison Matrix confirms zero regressions, the Config Diff highlights the exact 1-line change, and Llama 3.2 approves sign-off:

Approved MOP Run

Case B: Silent Misconfiguration Caught & Blocked

When a simulated mistake introduces ip mtu 1400 + no ip unreachables (or drops a BGP prefix / next-hop-self), the 1500B DF-bit probe immediately flags REGRESSION, the Unified Config Diff pinpoints the unauthorized lines, and Llama 3.2 rejects the change—activating a one-click rollback button:

Regression Caught

Audit-Ready Deliverables: 4-Sheet Excel & Markdown PIR

For CAB reviews and compliance audits, clicking Export Excel (.xlsx) in the top-right header generates a styled 4-tab workbook (Executive Summary, Pre-Post Matrix, Config Diffs, AI Audit Trace), alongside a timestamped Markdown Post-Implementation Review (reports/pir_*.md):

Excel Audit Report

Key Takeaways for SysOps & NetOps Engineers

1. Let Python do the math; let AI do the audit: Never rely on an LLM to diff raw configs when deterministic libraries (NAPALM, difflib) do it with 100% mathematical accuracy.

2. Enforce guardrails in code, not prompts: Prompt instructions can be bypassed or misread; Python allowlists cannot.

3. Audit trails drive adoption: Producing a clean .xlsx workbook and PIR Markdown file turns an experimental AI script into an operational tool CAB teams and managers can trust.

Try It Yourself (Open Source)

The complete source code, GNS3 baseline configs (R1.cfg–R3.cfg), and setup instructions are available on GitHub under the GNU GPLv3 license:

👉 github.com/shahkh-eng/netops-ai-mop-auditor

Clone the repository, boot the 3-router topology in GNS3, pull llama3.2 via Ollama, and test your own manual CLI changes in GNS3 to watch the auditor evaluate your work in real time. Contributions, new MOP scenarios, and feedback are always welcome!

References & Further Reading

  1. Network Programmability and Automation (2nd Edition) — Matt Oswalt, Christian Adell, Scott S. Lowe, and Jason Edelman (O'Reilly Media, 2023). An essential reference for Python, NAPALM, Netmiko, and structured network operations.
  2. Site Reliability Engineering: How Google Runs Production Systems — Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy (O'Reilly Media / sre.google). Invaluable guidance on change management, post-incident reviews, and mitigating automation risk.
  3. ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao et al. (arXiv:2210.03629). The foundational academic research behind tool-calling reasoning loops in LLM agents.
  4. RFC 1191 — Path MTU Discovery — J. Mogul and S. Deering (IETF). Detailed breakdown of the ICMP Type 3 Code 4 fragmentation signaling mechanism.
  5. RFC 4271 — A Border Gateway Protocol 4 (BGP-4) — Y. Rekhter, T. Li, and S. Hares (IETF). The core standard defining iBGP next-hop resolution and route advertisement rules.
  6. Cisco DevNet Code Exchange — developer.cisco.com/codeexchange. Community-driven repository for Cisco automation, APIs, and AI integrations.