Every network engineer knows the quiet tension of a 2:00 AM maintenance window.
You have reviewed your Method of Procedure (MOP), secured change-advisory board (CAB) approval, and executed configuration updates across core routers. To verify, you run show ip interface brief, confirm all interfaces are up/up, and send a standard 5-packet ICMP ping across the transit link:
Sending 5, 100-byte ICMP Echos to 172.16.30.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5) With five clean exclamation marks, everything appears healthy. You close your terminal, update the change ticket to "Completed," and log off.
Three hours later, morning business traffic ramps up, customer service lines light up, and the critical incident bridge convenes: enterprise web dashboards and database sessions between HQ and the Data Center are hanging and timing out.
What went wrong?
The control plane never dropped. OSPF remained in FULL state. BGP remained Established. Small 100-byte monitoring pings passed cleanly. However, an engineer had inadvertently committed ip mtu 1400 along with no ip unreachables on a core transit link while masking the mismatch using ip ospf mtu-ignore.
As soon as real application traffic arrived with 1500-byte frames and the Don't Fragment (DF) bit set, the packets were silently dropped. Because ICMP unreachables were disabled, Path MTU Discovery (PMTUD) failed silently.
Industry studies consistently reveal that over 70% of network outages are self-inflicted during scheduled maintenance windows. The root cause is rarely a complete physical link collapse; rather, it is subtle "gray failures" that easily slip past manual check sheets.
In this article, I walk through how I designed and built netops-ai-mop-auditor, an open-source, zero-trust Pre/Post Maintenance Verification & AI Change Auditor using Python, NAPALM, Netmiko, Flask, and local Llama 3.2 (via Ollama), validated end-to-end inside a GNS3 multi-AS Cisco IOS lab.
For Beginners: The Learning Path from Python Scripts to AIOps
If you are new to network automation—setting up virtual Cisco labs or writing your first Python connection scripts—AIOps becomes intuitive once the underlying mechanics are clear.
This project represents the practical evolution of a foundational multi-part automation series I previously published on LinkedIn. If you are starting from scratch or want to follow the progression step by step, start with these guides:
| Step | Topic / Focus | LinkedIn Walkthrough |
|---|---|---|
| 1. Lab Virtualization | GNS3 + VM Setup & Bridging | GNS3 2.1 GUI & VM Installation Steps — Setting up GNS3 and bridging virtual routers to your host machine via Cloud1 and Hub1. |
| 2. Python CLI Basics | Raw Socket Connections | Network Automation: Telnet to Cisco Router — Understanding programmatic command execution and terminal handling. |
| 3. Secure Automation | Enterprise SSH via Netmiko | Network Automation: SSH via Netmiko — Managing SSH sessions, handling command prompts, and navigating privilege levels. |
| 4. Multi-Vendor State | Structured Config with NAPALM | Network Automation: Configure Router with NAPALM — Moving beyond screen-scraping toward structured multi-vendor getters and config management. |
| 5. Web Dashboards | Telemetry via Nornir + Flask | Network Automation: Telemetry via Nornir & Flask — Surfacing live router metrics inside an interactive browser UI. |
Once you understand how to interact programmatically with network devices and render operational state on the web, the next engineering challenge is: How do we introduce Large Language Models (LLMs) into NetOps workflows safely without risking network downtime?
Why Naive "LLM Chatbots" Fail in Production NetOps
When engineers first experiment with AI, the common approach is to copy and paste hundreds of lines of show running-config before and after a change into an LLM with a prompt like:
"Here is the configuration before and after my change. Did anything break?"
In real-world network operations, this naive approach fails due to two distinct engineering hazards:
1. The Hallucination & Arithmetic Hazard
Small local models (such as 3B to 8B parameter models) are probabilistic language models, not deterministic calculation engines. When presented with 600 lines of unstructured Cisco IOS configuration text, they regularly miss subtle wildcard mask typos, miscalculate BGP prefix counts, or hallucinate non-existent interface names. Relying on an LLM to accurately diff routing tables in context memory is an architectural anti-pattern.
2. The Unacceptable Blast Radius of Autonomous Write Access
Granting an AI model autonomous write access (configure terminal) to enterprise core infrastructure is a risk no prudent engineering team will permit. A single prompt injection or hallucinated command syntax could trigger an unrecoverable routing loop or widespread blackout.
The Solution: The Anti-Hallucination Split Architecture
To make AI production-grade in networking, we enforce a strict separation of concerns:
- Deterministic Python Engine (
NAPALM+Netmiko+difflib): Machines excel at structured data collection, cryptographic snapshot hashing, mathematical line-by-line diffing, and synthetic data-plane probes. - Local Semantic Auditor (
Llama 3.2viaOllama): LLMs excel at reasoning over structured context, auditing semantic scope compliance against authorized human change requests (MOPs), and writing human-readable Post-Implementation Review (PIR) reports.
The GNS3 Lab Topology: Multi-AS Enterprise Core
The validation environment is hosted in GNS3 and simulates an enterprise-to-data-center transit architecture:
- R1 (
AS 65001– HQ Edge): Advertises HQ LAN192.168.10.0/24into eBGP acrossFa0/0 (10.12.0.1/30)to R2. - R2 (
AS 65002– Core ASBR): Bridges eBGP (AS 65001) to the internal core. Runs OSPF Area 0 onFa2/0 (10.23.0.1/30)to R3 and peers via iBGP (Loopback0: 2.2.2.2to3.3.3.3) applyingnext-hop-self. - R3 (
AS 65002– DC Edge): Advertises DC Server subnet172.16.30.0/24into BGP. - Out-of-Band (OOB) Management Plane: Interfaces
R1 e2/0 (172.20.40.211),R2 e1/0 (172.20.40.212), andR3 e1/0 (172.20.40.213)connect throughHub1to host loopback adapterCloud1 (172.20.40.0/24), completely isolated from data-plane traffic.
Zero-Trust Engineering Controls
To guarantee operational safety, the platform implements three strict design controls:
1. The 4-Layer Snapshot Engine (tools.py)
Every Pre-Check and Post-Check captures four distinct operational layers across R1, R2, and R3:
- Configuration Layer: Cleaned
running-config(volatile Cisco IOS byte-count headers and timestamps are stripped so clean diffs produce zero noise). - IGP Layer: OSPF neighbor Router-IDs and
FULLadjacency states. - BGP Overlay Layer: Peer session states, received prefix counts, and best-path (
*>) validity for critical HQ and DC prefixes. - Data Plane Layer: Synthetic
100-bytereachability pings and progressive1500-byte DF-bitPath MTU probes in both directions.
2. Regex-Enforced Read-Only Guardrails (guardrails.py)
The AI agent has zero write tools. Every CLI query requested by Llama 3.2 passes through strict Python validation that permits only show, ping, and traceroute commands while blocking shell escapes (;, | redirect) and write verbs (configure, reload, write, erase) using word-boundary regex patterns.
3. 100% Air-Gapped Local Inference
Running Llama 3.2 locally via Ollama ensures that router configurations, internal IP schemes, and operational telemetry never leave your local management perimeter.
The 4-Stage Maintenance Lifecycle in Action
The Flask web dashboard (featuring Dark/Light mode and real-time Server-Sent Events streaming) guides the engineer through a strict 4-step workflow:
- Select Maintenance Scenario (MOP) & Scope: Pick a planned activity (
MOP-101link hygiene,MOP-102new DC subnet onboarding,MOP-103iBGP tuning, orMOP-104custom manual GNS3 work) and define the Authorized Change Scope. - Run Pre-Checks (
PRE_ACTIVITY): Lock the 4-layer baseline. Once verified, the UI displays a green CLEARED banner authorizing the engineer to begin work. - Execute Activity (Manual GNS3 CLI or Simulation): Make real changes directly inside GNS3 consoles—or test with the built-in Clean vs. Misconfiguration simulators.
- Manual Engineer Confirmation → Post-Check & AI Verdict: When finished, the engineer clicks Confirm Changes Done. Python captures
POST_ACTIVITY, builds the Pre vs. Post Comparison Matrix and Unified Config Diff, and streams Llama 3.2's audit verdict.
Case A: Clean Maintenance Approved (MOP-101)
When only the authorized interface description on R2 FastEthernet2/0 is updated, the Comparison Matrix confirms zero regressions, the Config Diff highlights the exact 1-line change, and Llama 3.2 approves sign-off:
Case B: Silent Misconfiguration Caught & Blocked
When a simulated mistake introduces ip mtu 1400 + no ip unreachables (or drops a BGP prefix / next-hop-self), the 1500B DF-bit probe immediately flags REGRESSION, the Unified Config Diff pinpoints the unauthorized lines, and Llama 3.2 rejects the change—activating a one-click rollback button:
Audit-Ready Deliverables: 4-Sheet Excel & Markdown PIR
For CAB reviews and compliance audits, clicking Export Excel (.xlsx) in the top-right header generates a styled 4-tab workbook (Executive Summary, Pre-Post Matrix, Config Diffs, AI Audit Trace), alongside a timestamped Markdown Post-Implementation Review (reports/pir_*.md):
Key Takeaways for SysOps & NetOps Engineers
1. Let Python do the math; let AI do the audit: Never rely on an LLM to diff raw configs when deterministic libraries (NAPALM, difflib) do it with 100% mathematical accuracy.
2. Enforce guardrails in code, not prompts: Prompt instructions can be bypassed or misread; Python allowlists cannot.
3. Audit trails drive adoption: Producing a clean .xlsx workbook and PIR Markdown file turns an experimental AI script into an operational tool CAB teams and managers can trust.
Try It Yourself (Open Source)
The complete source code, GNS3 baseline configs (R1.cfg–R3.cfg), and setup instructions are available on GitHub under the GNU GPLv3 license:
👉 github.com/shahkh-eng/netops-ai-mop-auditor
Clone the repository, boot the 3-router topology in GNS3, pull llama3.2 via Ollama, and test your own manual CLI changes in GNS3 to watch the auditor evaluate your work in real time. Contributions, new MOP scenarios, and feedback are always welcome!
References & Further Reading
- Network Programmability and Automation (2nd Edition) — Matt Oswalt, Christian Adell, Scott S. Lowe, and Jason Edelman (O'Reilly Media, 2023). An essential reference for Python, NAPALM, Netmiko, and structured network operations.
- Site Reliability Engineering: How Google Runs Production Systems — Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy (O'Reilly Media / sre.google). Invaluable guidance on change management, post-incident reviews, and mitigating automation risk.
- ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao et al. (arXiv:2210.03629). The foundational academic research behind tool-calling reasoning loops in LLM agents.
- RFC 1191 — Path MTU Discovery — J. Mogul and S. Deering (IETF). Detailed breakdown of the ICMP Type 3 Code 4 fragmentation signaling mechanism.
- RFC 4271 — A Border Gateway Protocol 4 (BGP-4) — Y. Rekhter, T. Li, and S. Hares (IETF). The core standard defining iBGP next-hop resolution and route advertisement rules.
- Cisco DevNet Code Exchange — developer.cisco.com/codeexchange. Community-driven repository for Cisco automation, APIs, and AI integrations.




