The journey from "bag of words" to ChatGPT is really the story of one problem: how does a model remember and relate things across a sequence? This walks the whole arc with code — embeddings, RNNs/LSTMs, seq2seq, attention, transformers — and ends with a complete, runnable conversational-AI system: backend, memory, RAG grounding, streaming, and a chat UI.

Full original (canonical) on PrepStack: https://prepstack.co.in/blog/deep-learning-for-text-rnn-seq2seq-attention-transformers-chatbot-conversational-ai-project

TL;DR

  • Text needs sequence models because order and long-range context matter. Embeddings turn tokens into vectors; RNNs/LSTMs process them in order but forget over long distances.
  • Seq2Seq (encoder → decoder) maps an input sequence to an output sequence. The original version had a fatal bottleneck: cramming the whole input into one vector.
  • Attention fixed the bottleneck by letting the decoder look back at every input token. "Attention is all you need" then dropped the RNN entirely → the transformer.
  • Modern conversational AI = a pretrained transformer + memory + retrieval grounding + tools + guardrails. The model is one component, not the whole system.

1. Why text is hard for neural networks

A classifier sees a fixed-size input (an image is always 224×224). Text isn't: "ok" and a 5,000-word essay are both valid inputs. And meaning depends on order and distance — "the cat the dog chased ran" needs you to track who-did-what across many words.

Three problems to solve: representation (→ embeddings), sequence (→ RNNs → LSTMs → transformers), and long-range dependency (→ attention).

2. Embeddings — words become vectors

The first layer of every text model is an embedding: a lookup table mapping each token ID to a dense vector, learned so similar words land near each other.

import torch, torch.nn as nn embedding = nn.Embedding(10000, 128) token_ids = torch.tensor([[5, 42, 9, 1]]) # one sentence, 4 tokens vectors = embedding(token_ids) # shape (1, 4, 128) 

After training, embedding["king"] - embedding["man"] + embedding["woman"] ≈ embedding["queen"]. The network discovers semantic structure on its own.

3. RNNs process the sequence (but forget)

A Recurrent Neural Network reads tokens one at a time, maintaining a hidden state: h_t = f(h_{t-1}, x_t).

class RNNTextClassifier(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes): super().__init__() self.embed = nn.Embedding(vocab_size, embed_dim) self.rnn = nn.LSTM(embed_dim, hidden_dim, batch_first=True) self.fc = nn.Linear(hidden_dim, num_classes) def forward(self, x): emb = self.embed(x) out, (h_n, c_n) = self.rnn(emb) return self.fc(h_n[-1]) # classify from the last state 

A vanilla RNN forgets quickly — by token 100 the signal from token 1 has vanished (the vanishing gradient problem). LSTMs add gates (forget / input / output) to carry information across much longer spans. GRU is a lighter LSTM. They powered NLP ~2014–2018, but still process tokens sequentially and struggle with very long dependencies.

4. Sequence-to-Sequence

Many tasks map a sequence → another sequence: translation, summarization, dialogue, code generation. Seq2Seq uses an encoder that compresses the input into a context vector and a decoder that generates the output token by token.

class Encoder(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim): super().__init__() self.embed = nn.Embedding(vocab_size, embed_dim) self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True) def forward(self, x): emb = self.embed(x) outputs, (h, c) = self.lstm(emb) return outputs, (h, c) class Decoder(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim): super().__init__() self.embed = nn.Embedding(vocab_size, embed_dim) self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True) self.fc = nn.Linear(hidden_dim, vocab_size) def forward(self, token, hidden): emb = self.embed(token) out, hidden = self.lstm(emb, hidden) return self.fc(out), hidden 

The fatal flaw: the encoder squeezes the entire input into one fixed-size vector. For a 50-word paragraph that vector can't hold everything — early information gets crushed, and translation quality fell off a cliff as sentences got longer.

5. Attention and Transformers

Attention lets the decoder look at all the encoder's per-token states and decide, at each output step, which input tokens matter most.

import torch.nn.functional as F def attention(decoder_state, encoder_states): scores = encoder_states @ decoder_state # relevance of each input token weights = F.softmax(scores, dim=0) # turn into a distribution context = (weights.unsqueeze(-1) * encoder_states).sum(0) # weighted blend return context, weights # weights = interpretable alignment 

Then "Attention Is All You Need" (2017): if attention does the heavy lifting, drop the RNN entirely → the transformer. Two wins: parallelism (every token processed at once) and direct long-range connections (token 1 attends to token 500 in a single step).

Each token produces a Query, Key, and Value. Attention = match queries to keys, blend values.

import math, torch.nn.functional as F def self_attention(X, W_q, W_k, W_v): Q = X @ W_q; K = X @ W_k; V = X @ W_v scores = Q @ K.transpose(-2, -1) / math.sqrt(Q.size(-1)) weights = F.softmax(scores, dim=-1) # every token vs every token return weights @ V 

Three transformer shapes: encoder-only (BERT, understanding), decoder-only (GPT/Llama, generation + chat), encoder-decoder (T5/BART, seq2seq). Conversational AI uses decoder-only models.

6. Building Conversational AI

You do not train a chatbot from scratch. You take a pretrained instruct model and build a system around it. The model is one of five components:

  1. Memory — the model is stateless; feed it the conversation history every turn.
  2. Grounding (RAG) — retrieve relevant chunks and put them in the prompt. This stops it making things up.
  3. Prompt construction — system instructions + retrieved context + history + new message, via the model's chat template.
  4. The LLM — generates the reply.
  5. Guardrails — redact PII, block unsafe requests, cap tokens, rate-limit.

The conversation loop — retrieve, construct, generate, remember, trim — is the heart of every chatbot. The context window is a budget: trim with a sliding window, summarize old turns, or RAG over history.

7. The project

A complete chatbot: multi-turn memory, RAG grounding, streaming responses, guardrails, and a chat UI — hosted API or local model.

RAG retriever:

import numpy as np from sentence_transformers import SentenceTransformer class Retriever: def __init__(self, model_name="BAAI/bge-base-en-v1.5"): self.embedder = SentenceTransformer(model_name) self.chunks, self.embeddings = [], None def index(self, chunks): self.chunks = chunks self.embeddings = self.embedder.encode(chunks, normalize_embeddings=True) def search(self, query, k=4): if self.embeddings is None or not self.chunks: return [] q = self.embedder.encode([query], normalize_embeddings=True)[0] scores = self.embeddings @ q top = np.argsort(scores)[::-1][:k] return [self.chunks[i] for i in top] 

Pluggable streaming backend (OpenAI or local HF), FastAPI /chat with SSE, memory trimming to a token budget, PII redaction + safety guardrails, and a Gradio chat UI — the full code is in the original post.

Production hardening: real vector store (Qdrant/pgvector), persistent session storage, auth + rate limiting, tool/function calling, observability (Langfuse), evals on a golden Q&A set, and a smaller/cheaper model once quality is confirmed.

The arc in one picture

Bag of words → Embeddings → RNN/LSTM → Seq2Seq → + Attention → Transformer → Pretrained LLM → Conversational AI (LLM + memory + RAG + tools + guardrails) 

Every step solved a concrete limitation of the one before. Three habits: build the system, not the model; ground everything (RAG is the difference between a helpful assistant and a confident liar); stream, remember, and measure.

Originally published on PrepStack.__