How I Overhauled DocsGPT's Ingestion Engine & Supercharged RAG Context: My Hacktoberfest 2026 Journey
Hacktoberfest is often seen as a sprint to collect 4 quick pull requests. But this year, I wanted to do something different: dive head-first into one of the most prominent open-source AI knowledge bases—DocsGPT
—and tackle real, hard distributed systems and RAG optimization challenges that make a difference for thousands of production deployments.
Over the course of October, that mission turned into 3 major pull requests, thousands of lines of robust Python and TypeScript, a deep dive into Celery, Redis, and pgvector, and an incredible journey collaborating with DocsGPT maintainer Alex (@dartpain) and @coderabbitai.
Here is the deep-dive story of how I found the bugs, the architecture behind the fixes, how our ingestion overhaul (PR #3252) got merged into main, and what I learned along the way.
- The Mystery of the 24-Hour Source Lockout (PR #3068) The Sighting
It started with an issue report that sounded almost impossible: “When I add or remove files rapidly in a source, DocsGPT locks me out of my source for 24 hours!”
If you've ever worked on a document portal, you know that users don't wait patiently. They upload a PDF, realize they picked the wrong version, delete it, and re-upload. In DocsGPT, doing this triggered a phantom freeze. The UI spun indefinitely, and any attempt to re-sync returned deduplication errors.
Root Cause Investigation
Digging into the backend task dispatcher, I found the culprit in the Celery/Redis idempotency lock decorator.
To prevent duplicate re-ingestion jobs, DocsGPT synthesized an idempotency key with a static 24-hour Time-To-Live (TTL). The architectural flaw:
If a worker crashed, raised an uncaught network timeout to an embedding provider, or was terminated mid-task, the lock in Redis was never cleared.
Worse, on the frontend, the Server-Sent Events (SSE) listener was waiting for an active task ID. When a duplicate task was deduplicated into a sentinel object, the frontend listener stayed in an infinite wait loop—a classic poison-loop.
The Implementation Plan
Dynamic Lease Expiration: Switched the static 24h key to a heartbeated lease with automatic release on terminal worker failure.
Strict Task-ID Correlation: Updated FileTree and the SSE reingest pipelines to verify task ownership before blocking state transitions.
Poison-Loop Recovery: Ensured that sentinel deduplication events explicitly notify the frontend state machine to release the UI spinner gracefully.
- Ingestion Resumption, Orphan Cleanup & Real-Time Queue Telemetry (PR #3252 — Merged! 🎉)
While testing large docsets, another systemic bottleneck became obvious: long ingests.
When ingesting massive documentation portals (thousands of pages from a sitemap or hundreds of markdown files from a GitHub repository), two critical failure modes occurred:
The Worker Restart Death Sentence: If Celery was restarted (e.g., during a container redeploy or spot instance preemption) 90% of the way through a 2-hour ingest, the entire job was obliterated. All embedding tokens and compute spent were lost, forcing the user to start from scratch.
Orphaned Vector Pollution: If an ingest failed without a linked source entity, thousands of embedding chunks remained stranded in PostgreSQL (pgvector), silently degrading vector similarity search precision.
Designing the Solution
Together with the DocsGPT team, I designed a multi-layer ingestion reliability overhaul across 16 commits:
Database-Backed Checkpointing (0062_ingest_resume.py): Created a new IngestChunkProgress model and repository. As documents are parsed and vectorized in batches, chunk boundaries and offsets are recorded transactionally.
Reconciler Auto-Resume: Built an automated recovery hook in docsgpt/parser/ingest_resume.py. On worker boot, any interrupted task is inspected, its vectorstore progress verified, and ingestion resumes from the exact un-embedded offset.
Automatic Orphan Sweeper: Engineered a cleanup sweep in docsgpt/api/user/ingest_orphans.py that identifies unlinked vector embeddings in pgvector and purges them if an ingestion pipeline is aborted.
Real-Time Queue Depth Gauge (events/queue_gauge.py): Built an SSE-streamed queue depth and head-of-line wait time gauge so users can see exactly where their documents stand in the worker queue.
Result: Lead maintainer Alex (@dartpain) reviewed and merged PR #3252 directly into DocsGPT main!
[Merged] fix(ingest): resume, dedup, size caps, orphan cleanup and queue gauge for long ingests #3252
dartpain merged 16 commits into main from fix/ingest-reliability
- SuperCompress: Extractive Boilerplate Pruning for RAG (PR #3247)
With ingestion rock-solid, I tackled the next frontier in modern RAG: Context Window Efficiency (Issue #2578).
The Problem: Token Bloat in Retrieval
In enterprise RAG systems, retrieved documentation chunks are littered with noise:
Breadcrumb trails: Home > Products > Developer Settings > v1 > API
Cookie consent headers, footer links, and privacy disclaimers
Repeated markdown table headers across adjacent chunks
These lines consume 25% to 40% of the LLM prompt budget, pushing out real factual context and increasing inference latency and cost.
The Engineering Challenge
Naive regex or string stripping is dangerous in technical documentation:
If a user asks "How do I configure the API?", stripping Home > Products > API might remove the very subject of their search.
Stripping blank lines can break Markdown fenced code block syntax or remove four-space indentation on the first line of code.
Numeric comparison operators (value > 10) look suspiciously like breadcrumb separators (>) and can be falsely pruned!
The Architecture: CompressingRetriever
I built an extractive boilerplate pruning stage in docsgpt/retriever/stages/boilerplate_pruning.py and wrapped it cleanly in CompressingRetriever inside dispatcher.py:
python
class CompressingRetriever(BaseRetriever):
"""Delegating retriever that applies post-retrieval boilerplate pruning."""
def init(self, inner: BaseRetriever):
self._inner = inner
def search(self, *args, **kwargs) -> List[Dict[str, Any]]:
# Fall back to the inner retriever's stored question when search() receives no query
if args:
query = args[0]
elif "query" in kwargs:
query = kwargs["query"]
else:
query = (
getattr(self._inner, "original_question", None)
or getattr(self._inner, "question", None)
or ""
)
docs = self._inner.search(*args, **kwargs)
from docsgpt.retriever.stages.boilerplate_pruning import prune_boilerplate
return prune_boilerplate(docs, query=query or "")
Key features:
Fenced-Code Preservation: Carries active_fence state across chunk boundaries so continuation chunks don't have their code lines stripped as navigation.
Query-Aware Anchor Protection: Evaluates query terms before pruning; if a breadcrumb contains a user query term, it is protected.
Mathematical & Table Guards: Preserves pipe tables, indented code, and signed comparison expressions (x > 5).
- The Human-AI Review Dynamic: Shoutout to @dartpain & @coderabbitai
A huge part of why this Hacktoberfest was so rewarding was the review experience.
Maintainer Appreciation: Alex (@dartpain)
Maintaining a fast-moving, high-traffic repository like DocsGPT is demanding. Alex demonstrated what top-tier open-source stewardship looks like: prompt reviews, clear architectural expectations, and decisive merges when code met standards. His trust in letting me tackle deep parts of the ingestion engine was immensely motivating.
The Relentless Precision of @coderabbitai
DocsGPT employs CodeRabbit AI on every PR. At first, you might think an automated bot is just another linter. But CodeRabbit acted like an ultra-thorough senior engineer who never sleeps.
It caught subtle, high-impact edge cases that would have taken days of production debugging to surface:
Fenced-code continuation: Highlighting that continuation chunks inside a fenced code block without an opening fence could have unindented code mistakenly pruned.
Leading indentation retention: Catching that .strip() on a joined string stripped leading indentation from a first-line code block.
Stored question fallback: Pointing out that Dispatcher and ClassicRAG support search() using the question supplied at construction, meaning the wrapper needed to query self.original_question rather than passing an empty string.
Each piece of feedback became an opportunity to write cleaner, more defensive Python and back it with comprehensive unit tests.
- Key Takeaways for Open Source Contributors Find Real Pain Points: Don't just fix typos. Dig into open issues where users are hitting actual walls (like failed 2-hour ingests or locked sources). Write Tests for the Edge Cases You Didn't Think Of: Automated reviewers and CI checks are your friends. Every time a reviewer points out an edge case, write a dedicated test for it. Respect Maintainer Time: Write clean PR descriptions, link related issues, ensure DCO sign-offs (git commit -s), and keep your branches rebased against main.
Hacktoberfest 2026 has been an absolute blast. Huge thanks to the DocsGPT community, DigitalOcean, and MLH for organizing this global celebration of open source.
See you on the leaderboard, and happy hacking! 🚀
Follow my open-source work on GitHub: @satanic47
Portfolio & Live Demos: shivysidinfo.tech
