┌─────────────────────────────────────────────────────────────────────────────┐ │ THE "INFINITE CONTEXT" TRAP │ ├──────────────────────────────┬──────────────────────────────────────────────┤ │ 1. Attention Degradation │ Lost-in-the-Middle: critical interfaces get │ │ │ buried under repetitive DOM noise and loops. │ ├──────────────────────────────┼──────────────────────────────────────────────┤ │ 2. KV-Cache Prefill Lag │ Time-to-First-Token (TTFT) scales with prompt│ │ │ size; 150k+ raw tokens stall your agent. │ ├──────────────────────────────┼──────────────────────────────────────────────┤ │ 3. The "Tailwind Tax" │ Paying frontier API rates to ingest 80-char │ │ │ strings like "flex items-center justify-..." │ ├──────────────────────────────┼──────────────────────────────────────────────┤ │ 4. Rate-Limit Throttling │ Bloated prompts quickly exhaust TPM (Tokens │ │ │ Per Minute) quotas in CI/CD pipelines. │ └──────────────────────────────┴──────────────────────────────────────────────┘ Have you ever dumped an entire React or Next.js repository into Claude 3.5 Sonnet, GPT-4o, or a local Ollama model to ask: "How does authentication state flow through my UI, and what endpoints handle it?" If you inspect the prompt you sent, over 70% of the tokens are dead weight: Hundreds of lines of static Tailwind CSS utility classes (className="flex flex-col items-center justify-between p-8 bg-white dark:bg-zinc-950 rounded-2xl shadow-xl..."). Imperative loops, array mappers, and string formatting logic that have nothing to do with application architecture. Missing bird's-eye views: no clear backend API route table, no dependency graph, and no clean component hierarchy. While building an agentic Chrome extension powered by local LLMs, my context window collapsed: 209,757 tokens per scan. Responses took forever, local inference crawled, and the model routinely hallucinated core functions because key architectural interfaces were buried under syntactic noise. I built urai-ecma: a multi-threaded CLI tool written in Rust that uses SWC (Speedy Web Compiler) to parse JavaScript and TypeScript into Abstract Syntax Trees (AST). Instead of blindly concatenating files together like a text scraper, it acts as a semantic compiler for prompt engineering—compressing that same 209k token codebase down to 36k tokens (an 82.7% reduction) in milliseconds. Here is how it works, how it is architected under the hood, real benchmarks, and the engineering trade-offs you should know before using it. 🏛️ The Philosophy: What is "Urai"? In classical Tamil literary heritage, monumental masterworks like the Thirukkuṛaḷ (திருக்குறள்) and Tolkāppiyam (தொல்காப்பியம்) contained dense, multi-layered philosophical thought. To make these works practical without destroying their architectural depth, classical scholars practiced உரை எழுதுதல் (Urai Ezhuthudhal). Master commentators (Uraiyāsiriyars) like Parimelazhagar and Ilampuranar did not just copy or mechanically summarize texts. They performed structural distillation: Isolating the core semantic axioms of each stanza. Stripping linguistic ornamentation that obscured meaning. Exposing grammar, intent, and relationships for reasoned debate. Modern enterprise JavaScript and TypeScript codebases are the epic literatures of software engineering. When asking an LLM to reason about your code, it doesn't need raw syntactic exhaustion—it needs the structural anatomy, API contracts, state flows, and component signatures. urai-ecma acts as a modern Uraiyāsiriyar for your codebase. ⚡ Synthesis vs. Blind Concatenation Tools like repomix, gitingest, and code2prompt are file dumpers. They walk your directory, wrap raw text in XML/Markdown fences, and pass every single line of styling directly into your model's context. urai-ecma is an AST-aware compiler engine. Rather than treating code as raw strings, it parses your source into concrete syntax trees using ByteDance/Vercel’s swc_ecma engine and applies deterministic, semantic transformations: ┌─────────────────────────────────────────────────────────────────────────┐ │ URAI COMPILER PIPELINE │ └─────────────────────────────────────────────────────────────────────────┘ Enterprise Monorepo (.ts, .tsx, .js, .mjs, .json) │ ▼ [ignore::WalkBuilder (Rust)] Honor .gitignore, prune node_modules & dist │ ▼ [Rayon Parallel Work-Stealing] Multi-threaded AST parsing across all CPU cores │ ┌───────────────┴───────────────┐ ▼ ▼ [swc_ecma_parser] [swc_ecma_parser] Worker Thread A Worker Thread B │ │ ├─► [RouteVisitor] ├─► [RouteVisitor] │ Next.js/Express/NestJS │ Next.js/Express/NestJS │ │ ├─► [ReactComponentAnalyzer] ├─► [ReactComponentAnalyzer] │ Props, State, Hooks, JSX │ Props, State, Hooks, JSX │ │ ├─► [ReactJsxPruner] ├─► [ReactJsxPruner] │ Tailwind static class strip │ Tailwind static class strip │ │ └─► [FunctionSummarizerVisitor] └─► [FunctionSummarizerVisitor] Preserve structural stubs Preserve structural stubs │ ▼ [Foyer Hybrid Cache (Disk + RAM)] Sha512_256 + Zstd compression │ ▼ [swc_ecma_codegen + Tiktoken Engine] Emits high-density Markdown prompt + BPE o200k report 🎯 The Three Pillars of AST Token Optimization 1. AST Structural Stubbing (is_structural_stub_stmt) Traditional minification forces a bad compromise: either include full function bodies (wasting thousands of tokens on loops and math) or strip functions down to empty signatures (which deletes hooks, event listeners, and JSX layouts). urai-ecma solves this through Structural Stubbing. It inspects AST statements and retains only nodes critical to architectural comprehension: // Only statements defining component anatomy are preserved: fn is_structural_stub_stmt(stmt: &Stmt) -> bool { match stmt { Stmt::Decl(Decl::Fn(_)) => true, // Nested helper declarations Stmt::Decl(Decl::Var(var_decl)) => var_decl.decls.iter().any(|decl| { if let Some(init) = &decl.init { matches!(**init, Expr::Arrow(_) | Expr::Fn(_)) } else { false } }), Stmt::Expr(expr_stmt) => { if let Expr::Call(call_expr) = &*expr_stmt.expr && let Callee::Expr(callee_expr) = &call_expr.callee && let Expr::Ident(ident) = &**callee_expr { let name = ident.sym.as_ref(); // Preserves React Hooks, lifecycle timers, and global listeners: return name.starts_with("use") || name == "setTimeout" || name == "setInterval" || name.contains("addEventListener") || name.contains("requestIdleCallback"); } false } Stmt::Return(ret_stmt) => { // Preserves JSX layout hierarchies: if let Some(arg) = &ret_stmt.arg { matches!( &**arg, Expr::JSXElement(_) | Expr::JSXFragment(_) | Expr::Paren(_) ) } else { false } } _ => false, // Computational loops, arithmetic, & validations are pruned } } Hooks stay visible: useEffect(() => { ... }, [dep]) remains intact, signaling side-effects to the LLM. JSX hierarchies stay visible: The complete return layout tree remains legible. Imperative logic is replaced: Loops, arithmetic, and validation are replaced with a single synthetic docstring comment. 2. Dynamic-Aware Tailwind Pruning Modern utility CSS accounts for massive token bloat. urai-ecma provides 4 modes (remove, remove_aggr, summarize, preserve): Preserves Dynamic Expressions: If you use dynamic styling like className={clsx("btn", isActive && "btn-active")} or ternary conditions, they are kept 100% intact. Strips or Summarizes Static Strings: Long static class lists exceeding the threshold (default: 96 chars) are either cleanly stripped or sent to a local Ollama model to produce natural-language style summaries (e.g., /* UI: Frosted glass card with dark mode */). 3. JSDoc-First Resolution with Local Ollama Fallback Summarizing every single function with an LLM is slow. urai-ecma uses a two-tier resolution strategy: JSDoc First: It checks whether human-written JSDoc annotations (@description, @param, @return) already exist. It even includes a proximity-scan fallback (within a 300-byte span) to associate detached comments. This takes 0 milliseconds of LLM compute! Local Ollama Fallback: If a function exceeds your line threshold (default: 5 lines) and has no JSDoc, it queries a local Ollama model (gemma4, llama3.2). No code ever leaves your machine. Foyer Hybrid Caching: Results are stored in a dual-tier cache powered by Rust's foyer crate (64MB direct RAM buffer + 128MB Zstd-compressed disk storage with Sha512_256 keys). 🔍 Real-World Before & After Look at what happens to a bloated React component when passed through urai-ecma: ❌ The Bloated Source Code (Before) const ErrorUI = ({ headerDescTxt = "The real-time telemetry pipeline requires runtime binding. Ensure this window resides in a Chrome extension popup configured with permission parameters.", copyTextCommand = 'OLLAMA_ORIGINS="*" ollama serve', copyTagTxt = "MV3", copyHeaderTxt = "Manifest Interface Schema", copiedButtonTxt = "Copied Configuration", copyButtonTxt = "Copy Permission Manifest", }) => { const [copyState, setCopyState] = useState(false); const handleCopyManifest = () => { navigator.clipboard.writeText(copyTextCommand); setCopyState(true); setTimeout(() => setCopyState(false), 2000); }; const copyButtonTxtNode = copyState ? copiedButtonTxt : copyButtonTxt; return ( <div className="relative min-h-screen w-full bg-[#05050A] border border-white/10 overflow-hidden p-6 text-[#F8FAFC] flex flex-col justify-between"> <div className="absolute top-[-10%] left-[-10%] w-45 h-45 rounded-full bg-[#FF2E63] opacity-20 blur-[64px] pointer-events-none" /> <div className="space-y-6"> <div className="flex items-center space-x-3"> <div className="w-2.5 h-2.5 rounded-full bg-[#FF2E63] animate-pulse shadow-[0_0_8px_#FF2E63]" /> <span className="text-[10px] font-mono tracking-widest text-[#FF2E63] uppercase font-bold"> Diagnostics Status: Telemetry Offline </span> </div> <div className="bg-white/[0.03] border border-white/[0.08] rounded-2xl p-4 space-y-3"> <button className={`text-[9px] font-mono px-2 py-1 rounded-md transition-all bg-[#8B5CF6] text-white`}> {copyTagTxt} </button> <motion.button onClick={handleCopyManifest} className="w-full py-2 bg-white/[0.06] hover:bg-white/[0.1] border border-white/10 text-xs font-mono font-medium rounded-xl flex items-center justify-center space-x-2 text-white"> <span>{copyButtonTxtNode}</span> </motion.button> </div> </div> </div> ); }; ✅ Synthesized Output Generated by urai-ecma (After) ### React Component Breakdown: `<ErrorUI>` - **Props**: - `headerDescTxt` (type: `any`) [optional] - `copyTextCommand` (type: `any`) [optional] - `copyTagTxt` (type: `any`) [optional] - `copyHeaderTxt` (type: `any`) [optional] - `copiedButtonTxt` (type: `any`) [optional] - `copyButtonTxt` (type: `any`) [optional] - **State Management**: - Manages state `copyState` via setter `setCopyState`. - **Hooks**: Uses `useState` (Total Side-Effects: 0). - **Rendered JSX Tree**: `<div>, <span>, <h1>, <p>, <button>, <motion.button>` const ErrorUI = ({ headerDescTxt = "...", copyTextCommand = "...", copyTagTxt = "MV3", copyHeaderTxt = "...", copiedButtonTxt = "...", copyButtonTxt = "..." })=>{ const handleCopyManifest = ()=>{ setTimeout(()=>setCopyState(false), 2000); '/* "Copies the manifesto text to the clipboard and sets a temporary success state for two seconds." */'; }; return ( <div className="/* UI: Full-screen dark mode layout with border and text overflow management */"> <div className="/* UI: Absolute background blur element positioned outside the main container */"/> <div> <span>Diagnostics Status: Telemetry Offline</span> <div> <button className={`text-[9px] font-mono px-2 py-1 rounded-md transition-all bg-[#8B5CF6] text-white`}> {copyTagTxt} </button> <motion.button onClick={handleCopyManifest}> <span>{copyButtonTxtNode}</span> </motion.button> </div> </div> </div> ); '/* "Presents a user interface displaying diagnostic status and provides a copy function for a necessary runtime binding command." */'; }; Notice what happened: Zero structural loss: The LLM sees the destructured props, default parameters, state variables, setters, and timers. Dynamic styles preserved: Dynamic template literals like `text-[9px] ... ${...}` were left untouched. Static noise eliminated: Over 200 characters of utility noise were converted into short structural summaries. 📊 Benchmarks: Cold Runs vs. Warm Cache Runs We benchmarked urai-ecma on an Apple Silicon machine across different workloads using OpenAI’s native o200k_base BPE tokenizer: Benchmark 1: Real-World Monorepo Monolith (Chrome Agentic AI Extension) Metric Raw Project (TS/TSX) urai-ecma Output Total Reduction Token Volume (o200k_base) 209,757 tokens 36,153 tokens -82.76% 🚀 Downstream LLM Context Exceeds local 64k limits Fits easily in local Ollama Usable on 8GB VRAM KV-Cache TTFT (Time-to-First-Token) ~18.4 seconds ~1.9 seconds ~9.6x Faster Benchmark 2: Tactical Project (plugin-api-docgen) — Cold vs. Warm Performance Here are the terminal runs comparing an initial cold run (querying local Ollama) against subsequent warm runs (hitting the foyer hybrid cache): # RUN 1: Cold Execution (AST Parsing + Local Ollama Inference) $ time urai-ecma 🚀 [urai-ecma] Starting AST Analysis on project: ./src 🔍 Found 6 source file(s) for analysis. ✅ [urai-ecma] Prompt successfully generated at: ./output.md 📊 [urai-ecma] Estimated Tokens in ./output.md: 1317 tokens ============================================================ 📊 TOKEN SAVINGS & OPTIMIZATION REPORT ============================================================ 📁 Raw Source Code (All JS/TS): 3023 tokens ⚡ Optimized Output (output.md): 1317 tokens ------------------------------------------------------------ 🎉 Reduction: -56.43% tokens saved! (Saved ~1706 tokens) ============================================================ urai-ecma 0.06s user 0.04s system 0% cpu 21.541 total # RUN 2: Warm Execution (AST Parsing + Foyer Hybrid Cache Hits) $ time urai-ecma 🚀 [urai-ecma] Starting AST Analysis on project: ./src 🔍 Found 6 source file(s) for analysis. ✅ [urai-ecma] Prompt successfully generated at: ./output.md 📊 [urai-ecma] Estimated Tokens in ./output.md: 1299 tokens ============================================================ 📊 TOKEN SAVINGS & OPTIMIZATION REPORT ============================================================ 📁 Raw Source Code (All JS/TS): 3023 tokens ⚡ Optimized Output (output.md): 1299 tokens ------------------------------------------------------------ 🎉 Reduction: -57.03% tokens saved! (Saved ~1724 tokens) ============================================================ urai-ecma 0.03s user 0.01s system 92% cpu 0.049 total Performance Breakdown Execution Latency Comparison (plugin-api-docgen) ────────────────────────────────────────────────────────────────── Cold Run (Ollama Local Inference): ████████████████████ 21.541s Warm Run (Foyer Zstd Cache): ▍ 0.049s (49ms) ────────────────────────────────────────────────────────────────── Speedup Factor: ~439x Faster on Cache Hit! Cold Run (21.5s): The tool parses the AST in under 15ms, but waits on the local Ollama daemon to infer function docstrings sequentially. Warm Run (49ms - 68ms): On subsequent runs, Sha512_256 keys hit memory/disk cache entries with sub-millisecond latency. The entire repository is analyzed and emitted in less than 50 milliseconds! 🛠️ Tactical Comparison: urai-ecma vs. Existing Tools Feature repomix / gitingest code2prompt urai-ecma Engine Node.js / Python Rust Rust (SWC + Rayon) Parsing Strategy Naive String Concatenation Handlebars Templates True AST Traversal Tailwind Handling Preserves all noise (0% saved) Preserves all noise (0% saved) 4-Mode AST Pruning Structural Stubbing ❌ No ❌ No ✅ Yes (is_structural_stub) API Route Tables ❌ No ❌ No ✅ Auto-extracted (Next/Nest/Express) Component Analysis ❌ No ❌ No ✅ Props, State, Hooks, JSX tree Privacy / Offline Dependent on API Offline string copy 100% Offline (Local Ollama/JSDoc) Token Savings 0% (Expands token size) 0% Up to 82.7% reduction ⚠️ Brutal Honesty: Trade-offs & When NOT to Use It No tool is a silver bullet. Because urai-ecma prunes function internals into architectural stubs, you need to understand when to use it and when to skip it: ❌ When NOT to Use urai-ecma Algorithmic Debugging: If you are trying to find an off-by-one bug in a sorting algorithm or a matrix multiplication loop, the function body is pruned. An LLM cannot debug math it cannot see! Memory Leak Profiling: If you need an LLM to inspect improper closure references or un-cleaned subscriptions inside an imperative block, you need raw source files. Writing Low-Level Unit Tests: If your unit tests need to mock intermediate local variables inside a 50-line imperative function, you need the full implementation. ✅ When urai-ecma Shines Architectural Audits & Refactoring: High-level reviews of Next.js, React, or Express architectures. Autonomous AI Coding Agents (Claude Code, Aider, Cursor): Keeping an agent's scratchpad clean so it navigates files without blowing through token limits. Spec-Driven Scaffolding: Passing an entire repository's contracts and interfaces to an LLM to generate fresh implementations. Technical Interview Preparation: Asking questions about complex repositories without hallucinated paths or interfaces. Cross-Language Migrations (TypeScript $\to$ Rust/Go): Feeding the pure functional contract into an LLM without "JavaScript-isms" leaking into the generated target language. 🚀 Installation & Quick Start urai-ecma is distributed as a single static binary with zero runtime dependencies. 🐧 macOS & Linux (Shell) curl --proto '=https' --tlsv1.2 -LsSf https://github.com/sanjaiyan-dev/urai-ecma/releases/download/v0.1.1/urai-ecma-installer.sh | sh 🪟 Windows (PowerShell) powershell -ExecutionPolicy Bypass -c "irm https://github.com/sanjaiyan-dev/urai-ecma/releases/download/v0.1.1/urai-ecma-installer.ps1 | iex" 📦 Node.js (Global Package Managers) npm install -g urai-ecma # or: pnpm add -g urai-ecma | bun add -g urai-ecma | deno add -g npm:urai-ecma 🦀 Rust (Cargo) cargo install urai-ecma ⚙️ Configuration & SchemaStore Support Initialize a documented configuration file in your project root: urai-ecma create This creates urai.config.jsonc. Because urai-ecma is registered globally with SchemaStore, you get instant auto-complete and documentation in VS Code, WebStorm, IntelliJ, and Visual Studio: { "$schema": "https://www.schemastore.org/urai-ecma.json", // Source directory or file to analyze "input_project": "./src", // Target markdown prompt output "output_file": "./output.md", // Local Ollama instance (Optional) "ollama_endpoint": "http://localhost:11434", "ollama_modelname": "gemma4", // Tailwind CSS mode: "remove" | "remove_aggr" | "summarize" | "preserve" "tailwind_mode": "remove", "tailwind_threshold": 96, // Summarize function bodies via JSDoc or Ollama "summarize_functions": true, "summarize_functions_threshold": 5, // Extract Express / Fastify / Next.js / NestJS routes "generate_route_table": true, // React component introspection (Props, State, Hooks) "analyze_react_components": true, // Generate ASCII tree & Mermaid ESM dependency graph "generate_file_graph": true } Run analysis anywhere: # Run with configuration file urai-ecma # Or run ad-hoc via CLI flags urai-ecma -i ./src -o prompt.md --tailwind-mode remove 🔗 Try It Out & Contribute 📖 Documentation: https://sanjaiyan-dev.github.io/urai-ecma 🐙 GitHub Repository: https://github.com/sanjaiyan-dev/urai-ecma 🤖 For AI Agents (llms.txt): https://sanjaiyan-dev.github.io/urai-ecma/llms-full.txt sanjaiyan-dev / urai-ecma AST-aware JS/TS codebase-to-prompt compiler in Rust. Uses SWC & Rayon to prune Tailwind bloat, retain structural stubs, extract Next.js/Nest routes, and compress repos into hyper-dense prompts for GPT-4o, Claude 3.5 & Ollama. Benchmarked with Tiktoken o200k for 80%+ token savings. 🏛️ URAI (உரை) AST-Aware JS/TS Codebase-to-Prompt Engine for LLMs Transform bloated JavaScript & TypeScript repositories into hyper-dense, token-optimized context prompts. 🏛️ The Name Inspiration: The Art of "உரை எழுதுதல்" (Urai Ezhuthudhal) In classical Tamil literary heritage, monumental epics and ancient treatises—such as the Thirukkuṛaḷ, Tolkāppiyam, and Cilappatikāram—span vast volumes of dense, poetic, and complex thought. To make these monumental texts intelligible without losing their depth, classical scholars (Uraiyāsiriyars) practiced உரை எழுதுதல் (Urai Ezhuthudhal): the disciplined art of writing a lucid, structured, and insightful commentary that distills the core essence, syntax, and architectural meaning of vast literature. The Modern Parallel Today, enterprise JavaScript and TypeScript codebases are the epic literatures of modern software. Spanning thousands of files across Next.js, React, Node.js, and TypeScript, they are laden with boilerplate, repetitive utility classes, and nested syntax. When feeding these systems to Large Language Models: … View on GitHub If you're sick of burning through API credits and watching your AI coding agents drown in static CSS strings, give urai-ecma a spin on your project. Drop your before-and-after token savings in the comments below!