The journey from "bag of words" to ChatGPT is really the story of one problem: how does a model remember and relate things across a sequence? This walks the whole arc with code — embeddings, RNNs/LSTMs, seq2seq, attention, transformers — and ends with a complete, runnable conversational-AI system: backend, memory, RAG grounding, streaming, and a chat UI.
Full original (canonical) on PrepStack: https://prepstack.co.in/blog/deep-learning-for-text-rnn-seq2seq-attention-transformers-chatbot-conversational-ai-project
TL;DR
- Text needs sequence models because order and long-range context matter. Embeddings turn tokens into vectors; RNNs/LSTMs process them in order but forget over long distances.
- Seq2Seq (encoder → decoder) maps an input sequence to an output sequence. The original version had a fatal bottleneck: cramming the whole input into one vector.
- Attention fixed the bottleneck by letting the decoder look back at every input token. "Attention is all you need" then dropped the RNN entirely → the transformer.
- Modern conversational AI = a pretrained transformer + memory + retrieval grounding + tools + guardrails. The model is one component, not the whole system.
1. Why text is hard for neural networks
A classifier sees a fixed-size input (an image is always 224×224). Text isn't: "ok" and a 5,000-word essay are both valid inputs. And meaning depends on order and distance — "the cat the dog chased ran" needs you to track who-did-what across many words.
Three problems to solve: representation (→ embeddings), sequence (→ RNNs → LSTMs → transformers), and long-range dependency (→ attention).
2. Embeddings — words become vectors
The first layer of every text model is an embedding: a lookup table mapping each token ID to a dense vector, learned so similar words land near each other.
import torch, torch.nn as nn embedding = nn.Embedding(10000, 128) token_ids = torch.tensor([[5, 42, 9, 1]]) # one sentence, 4 tokens vectors = embedding(token_ids) # shape (1, 4, 128) After training, embedding["king"] - embedding["man"] + embedding["woman"] ≈ embedding["queen"]. The network discovers semantic structure on its own.
3. RNNs process the sequence (but forget)
A Recurrent Neural Network reads tokens one at a time, maintaining a hidden state: h_t = f(h_{t-1}, x_t).
class RNNTextClassifier(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes): super().__init__() self.embed = nn.Embedding(vocab_size, embed_dim) self.rnn = nn.LSTM(embed_dim, hidden_dim, batch_first=True) self.fc = nn.Linear(hidden_dim, num_classes) def forward(self, x): emb = self.embed(x) out, (h_n, c_n) = self.rnn(emb) return self.fc(h_n[-1]) # classify from the last state A vanilla RNN forgets quickly — by token 100 the signal from token 1 has vanished (the vanishing gradient problem). LSTMs add gates (forget / input / output) to carry information across much longer spans. GRU is a lighter LSTM. They powered NLP ~2014–2018, but still process tokens sequentially and struggle with very long dependencies.
4. Sequence-to-Sequence
Many tasks map a sequence → another sequence: translation, summarization, dialogue, code generation. Seq2Seq uses an encoder that compresses the input into a context vector and a decoder that generates the output token by token.
class Encoder(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim): super().__init__() self.embed = nn.Embedding(vocab_size, embed_dim) self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True) def forward(self, x): emb = self.embed(x) outputs, (h, c) = self.lstm(emb) return outputs, (h, c) class Decoder(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim): super().__init__() self.embed = nn.Embedding(vocab_size, embed_dim) self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True) self.fc = nn.Linear(hidden_dim, vocab_size) def forward(self, token, hidden): emb = self.embed(token) out, hidden = self.lstm(emb, hidden) return self.fc(out), hidden The fatal flaw: the encoder squeezes the entire input into one fixed-size vector. For a 50-word paragraph that vector can't hold everything — early information gets crushed, and translation quality fell off a cliff as sentences got longer.
5. Attention and Transformers
Attention lets the decoder look at all the encoder's per-token states and decide, at each output step, which input tokens matter most.
import torch.nn.functional as F def attention(decoder_state, encoder_states): scores = encoder_states @ decoder_state # relevance of each input token weights = F.softmax(scores, dim=0) # turn into a distribution context = (weights.unsqueeze(-1) * encoder_states).sum(0) # weighted blend return context, weights # weights = interpretable alignment Then "Attention Is All You Need" (2017): if attention does the heavy lifting, drop the RNN entirely → the transformer. Two wins: parallelism (every token processed at once) and direct long-range connections (token 1 attends to token 500 in a single step).
Each token produces a Query, Key, and Value. Attention = match queries to keys, blend values.
import math, torch.nn.functional as F def self_attention(X, W_q, W_k, W_v): Q = X @ W_q; K = X @ W_k; V = X @ W_v scores = Q @ K.transpose(-2, -1) / math.sqrt(Q.size(-1)) weights = F.softmax(scores, dim=-1) # every token vs every token return weights @ V Three transformer shapes: encoder-only (BERT, understanding), decoder-only (GPT/Llama, generation + chat), encoder-decoder (T5/BART, seq2seq). Conversational AI uses decoder-only models.
6. Building Conversational AI
You do not train a chatbot from scratch. You take a pretrained instruct model and build a system around it. The model is one of five components:
- Memory — the model is stateless; feed it the conversation history every turn.
- Grounding (RAG) — retrieve relevant chunks and put them in the prompt. This stops it making things up.
- Prompt construction — system instructions + retrieved context + history + new message, via the model's chat template.
- The LLM — generates the reply.
- Guardrails — redact PII, block unsafe requests, cap tokens, rate-limit.
The conversation loop — retrieve, construct, generate, remember, trim — is the heart of every chatbot. The context window is a budget: trim with a sliding window, summarize old turns, or RAG over history.
7. The project
A complete chatbot: multi-turn memory, RAG grounding, streaming responses, guardrails, and a chat UI — hosted API or local model.
RAG retriever:
import numpy as np from sentence_transformers import SentenceTransformer class Retriever: def __init__(self, model_name="BAAI/bge-base-en-v1.5"): self.embedder = SentenceTransformer(model_name) self.chunks, self.embeddings = [], None def index(self, chunks): self.chunks = chunks self.embeddings = self.embedder.encode(chunks, normalize_embeddings=True) def search(self, query, k=4): if self.embeddings is None or not self.chunks: return [] q = self.embedder.encode([query], normalize_embeddings=True)[0] scores = self.embeddings @ q top = np.argsort(scores)[::-1][:k] return [self.chunks[i] for i in top] Pluggable streaming backend (OpenAI or local HF), FastAPI /chat with SSE, memory trimming to a token budget, PII redaction + safety guardrails, and a Gradio chat UI — the full code is in the original post.
Production hardening: real vector store (Qdrant/pgvector), persistent session storage, auth + rate limiting, tool/function calling, observability (Langfuse), evals on a golden Q&A set, and a smaller/cheaper model once quality is confirmed.
The arc in one picture
Bag of words → Embeddings → RNN/LSTM → Seq2Seq → + Attention → Transformer → Pretrained LLM → Conversational AI (LLM + memory + RAG + tools + guardrails) Every step solved a concrete limitation of the one before. Three habits: build the system, not the model; ground everything (RAG is the difference between a helpful assistant and a confident liar); stream, remember, and measure.
Originally published on PrepStack.__
