Beyond RAG: The Evolution of AI Knowledge & Memory : From LLMs to Vectorless RAG, LLM Wiki, and Memory

So recently I have been learning a lot about AI systems and architecture and how to build scalable AI pipelines that you can ship into production.

But I haven't seen a single blog that covers all these topics together. So let's start!

Topics We cover in this blog :

1. LLMS
2. Vector Rag
3. Vectorless Rag
4. LLM Wiki
5. MeM0 or Memory in Ai agents

LLMs

LLM stands for Large Language Models. They are trained on large amounts of data to predict the next token based on the tokens they receive.

Flow

The user query (input tokens) is converted into embeddings that capture semantic information. Then the Transformer (a machine you can say) uses attention to understand the context and calculates a probability distribution for the next token.

Example

 Input: What is the capital of India? The LLM processes the input and predicts tokens one by one based on probability. Example flow: 1. The 2. capital 3. of 4. India 5. is 6. New 7. Delhi 8. . Final output: The capital of India is New Delhi. 

Each new token is predicted using the original prompt + all previously generated tokens.

Problems with LLMs

LLMs are good for general-purpose tasks. If you are asking normal daily questions then they work just fine.

But the real problem starts when you add a large document such as a 3000-page PDF at once. You may hit the token limit quickly or get poor-quality answers because of the context window limitation.

Context Window: The maximum number of tokens an LLM can process at once as input and generate as output.

So how do we fix this?

Here comes RAG. Let's start!

Btw i have a detailed blog on RAG and How a basic RAG pipeline looks you can check this out 👇

Where RAG Fails: Understand the Limitations

Kunal

Kunal

Kunal

Follow

Jul 18

Where RAG Fails: Understand the Limitations

#ai #programming #learning #deeplearning

5 reactions

Add Comment

5 min read

RAG

RAG stands for Retrieval-Augmented Generation. It retrieves relevant information from an external knowledge source (the documents you upload) and provides it to the LLM along with the user's question.

The problem with a plain LLM is that every model has a knowledge cutoff date. It relies on the knowledge it learned during training.

Now imagine uploading thousands of PDFs and asking questions about them. The LLM has to read all those documents every time you ask a question. This takes more tokens and fills up the context window quickly. As a result you get higher costs and sometimes poor answers.

So how do we solve this problem? RAG comes into the picture.

How RAG works ?

RAG has two major phases :

Indexing :- Uploading the document
Query :- Asking the question to LLM from that uploaded document

Indexing

Indexing basically means converting your files into embeddings and storing them inside a vector database.

There are some steps:

1. Upload your document: Upload the document you want to work with.
2. Text Extraction: Extract text from the uploaded document. (Here we are talking about normal text-based documents. For images or scanned PDFs we need OCR.)
3. Chunking: Divide the extracted text into smaller chunks.
4. Embedding: Convert these chunks into numerical vectors that represent their semantic meaning here you need an embedding model.
5. Vector Database: Store these embeddings along with their associated text chunks and metadata.

Query

Query simply means asking a question to your RAG based on the files you have already uploaded.

PS: Image credit goes to LangChain. Source: LangChain Documentation

There are some steps:

1. Ask a Question: The user enters a query.
2. Embedding: Convert the query into a numerical vector that represents its semantic meaning.
3. Similarity Search: Search the vector database to find chunks that are semantically similar to the query.
4. Top-K Results: Retrieve only the top-K most relevant chunks to avoid passing unnecessary information to the LLM.

This is how traditional RAG works. It solves many problems but also has some limitations. Let's first understand what it solves and then discuss the problems with traditional RAG.

What problems does RAG solve?

With RAG you can:

1. Give the LLM access to your personal documents or external knowledge that it was not trained on.
2. Generate answers based on the relevant information retrieved from your documents.

NOTE: RAG does not increase the context window. It simply reduces the amount of information that needs to be passed into the context window.

The Problems with Traditional RAG

Traditional RAG solves many problems but it also introduces some new challenges.

1. Retrieval Quality

Imagine you want to find something in your document but you don't know its exact definition. You only understand the concept.

For example you want to find information about the Temporal Dead Zone in an uploaded document. You describe the concept to the LLM but the retrieval system fails to find the relevant chunks because your query does not match the document content properly.

Bad Input = Bad Output.

2. Chunking Problem

Chunking plays an important role because embeddings are generated from these chunks.

Imagine you have this document:

Our company provides a salary of ₹50,000. Employees are eligible for a yearly performance bonus. The bonus depends on employee performance. 

Now imagine your chunking process divides it into:

Chunk 1: Our company provides a salary of ₹50,000. Chunk 2: Employees are eligible for a yearly performance bonus. Chunk 3: The bonus depends on employee performance. 

Now the user asks:

How much salary does an employee receive and are they eligible for a bonus?

The retrieval system might retrieve Chunk 1 and Chunk 2 but miss Chunk 3.

As a result the LLM gets incomplete information and may generate an incomplete answer.

The problem is that bad chunking can lead to missing context and poor retrieval quality.

1. Lost relationships : A chunk may contain a reference or relation ship to another paragraph which might lost in the chunking process

NOTE: Chunking is not a flaw in RAG. However it is still a major challenge because there is no fixed rule for how to chunk a document or how large each chunk should be. If chunking is not done properly we may lose important context and relationships between chunks.

3. Semantic Retrieval Is Not Always Accurate

Semantic search is not always accurate because it retrieves results based on the similarity of embeddings.

Sometimes words can have multiple meanings or different words can have the same meaning.

For example:

Example 1: Same word with different meanings

User: I want to go to the bank. Document: A bank of money and a bank of a river. 

Example 2: Different words with the same meaning

User: Find employee salary information. Document: Monthly compensation details. 

In these cases the retrieval system may fail to find the correct information or retrieve irrelevant chunks.

As a result the LLM may receive incorrect or incomplete context and generate a wrong answer or hallucinate.

This brings us to alternative retrieval approaches.

Vectorless RAG

Vectorless RAG is a RAG approach that retrieves information without relying on vector embeddings. It uses traditional information retrieval techniques such as Keyword matching , full text search and some keyword searching algorithm.

How it works ?

There are some steps:

1. Documents: Upload the documents you want to search.
2. Text Processing: Extract the text from the uploaded documents.
3. Inverted Index: Extract keywords from the documents and map them to the documents where they appear.
4. BM25 / Keyword Search: Search for matching keywords from the user's question in the inverted index.
5. Relevant Documents: Get the most relevant documents based on the matching keywords.
6. LLM: Pass the relevant documents along with the user's question to the LLM to generate the final answer.

Does Vectorless RAG solve chunking?

No , Vectorless rag just removes the dependency on dense embeddings based retrieval you can use both retrieval at the same time , but it does not automatically eliminate chunking.

 User Query | +-------+-------+ | | v v BM25 Search Vector Search | | v v Keyword Semantic Results Results | | +-------+-------+ | v Score Fusion | v Reranker | v Relevant Context | v LLM 

This above is known as the Hybrid retrieval we use it when we need both exact matching and meaning based retrieval.

Vector Search vs Vectorless Search

Now do you notice something Traditional RAG retrieves information from raw documents every time a user asks a question.

It solves the LLM problem but dont you think system is just effectively repeats a lot of knowledge discovery.

What if we could read and organize the documents once, and maintain the extracted knowledge in a structured knowledge base?

This is the idea behind an LLM Wiki.

LLM Wiki

An LLM Wiki is a knowledge base in which an LLM reads raw documents, creates structured interconnected knowledge pages, and continuously updates those pages as new information arrives.

It's just like creating your own notes to help you revise everything a day before your exams.

Why Does LLM Wiki Come Into the Picture?

Consider this example.

You are studying GenAI and have collected several PDFs:

raw/ rag.pdf embeddings.pdf vector-databases.pdf llm-agents.pdf memory-systems.pdf 

Instead of searching through these PDFs every time you ask a question the LLM reads them and creates its own wiki using Markdown files.

 RAW SOURCES | v Text Extraction | v Information Processing | v LLM Analysis | v Create / Update Knowledge Pages | v KNOWLEDGE WIKI | +--------+--------+ | | | v v v RAG.md Agents.md Memory.md | | | +--------+--------+ | v Cross-linked Knowledge Base 

The LLM does not just create separate summaries of each PDF. It also connects related topics and identifies how different concepts are connected.

Now when you ask:

Explain how RAG evolved and why Vectorless RAG was introduced.

The LLM can use its already-organized knowledge instead of searching through all the raw PDFs again.

How LLM Wiki Works

It generally has three main operations:

A. Ingest

You upload a new research paper.

The LLM:

1. Reads the document.
2. Extracts important information.
3. Creates a summary of the document.
4. Updates related knowledge pages.
5. Updates the wiki index.
6. Checks if the new information conflicts with existing knowledge.

B. Query

You ask a question.

The LLM:

1. Checks the wiki index.
2. Finds the relevant knowledge pages.
3. Reads related information from those pages.
4. Generates an answer with references.
5. Can save useful new information back into the wiki.

C. Lint

The system checks the wiki for problems.

It:

1. Finds broken links between pages.
2. Checks for duplicate information.
3. Finds outdated or conflicting information.
4. Checks whether the knowledge pages are properly connected.

This helps keep the wiki clean and organized.

RAG vs LLM Wiki

Does LLM Wiki eliminate RAG?

No.
It changes how knowledge is organized and accessed.
You can still use keyword search, vector search, or other retrieval methods to find relevant wiki pages.

Memory in Agents

Now we have covered about LLMs and How an LLM can accesses an external knowledge

But do you know that LLMs are stateless? It means they don't have memory by default. We have to add a memory system so they can remember what we asked them before.

for example

Day 1: User: I am a final-year B.Tech student. I am learning Generative AI. I prefer short explanations. AI: Okay. 

Now the user closes the application.

After a few days, the user opens a new conversation:

Day 5: User: Tell me something about me AI: I dont know about you 

What can we do here? We can simply add memory. But just adding memory is not the solution. It creates some problems:

• Increased token consumption.
• Larger prompt size.
• Irrelevant conversation history.
• Difficulty managing long-term user information.

Solution: We can add an AI memory layer that stores only relevant information and retrieves it whenever needed.

Tools like Mem0 provide this memory layer. You can also use other memory solutions that offer similar functionality.

It allows AI agents to remember important facts across conversations without sending the entire conversation history every time.

Thanks for reading ! if enjoyed this blog , you can read more on this 👇

Kunal Follow

My tokens expired idk how to write bio. 😪🫠