"Generative AI" is on every roadmap, but most introductions either drown you in math or hand-wave with "it predicts the next word." Neither helps you build something or reason about when it's the wrong tool. This is a developer's on-ramp: model families, GPT & transformers, language modeling, the ethics you must handle, and a complete runnable text-gen app. (Full version on the original — link at the end.)

TL;DR

  • Generative models learn to produce data (P(data)); discriminative models only label it (P(label | input)). "Draw me a cat" vs "is this a cat?"
  • Four families: autoregressive (LLMs/GPT — text), diffusion (images/video/audio), GANs (fast narrow images), VAEs (latent spaces). For text, autoregressive transformers dominate.
  • GPT = a decoder-only transformer trained to predict the next token. Generation is a loop: predict → sample → append → repeat.
  • Language modeling is the objective; text generation is the application. Decoding (temperature, top-p) shapes output as much as the model.
  • Ethics is not optional — hallucination, bias, copyright, privacy, misuse, and cost are shippable risks. Handle them at the boundary.

1. What "generative" actually means

You already know discriminative models — they draw boundaries between classes ("spam or not spam") and learn P(label | input). Generative models learn the distribution of the data itself — P(data) — so they produce new samples that look like the training data.

Discriminative: input -> label (classify, score, detect) Generative: prompt -> new data (write, draw, compose, synthesize) 

The practical consequence: there's no single "right answer." Given "Once upon a time," there are billions of valid continuations. That's both the power (creativity) and the danger (confident, plausible nonsense).

2. The four families (and a picking guide)

Not all generative AI is an LLM. Knowing the families is how you pick the right tool:

  • Autoregressive (the LLM family) — generate one token at a time, each conditioned on everything before it: P(x1..xn) = Π P(xi | x<i). GPT, Llama, Claude. Best for: text, code, any sequence.
  • Diffusion — start with noise and iteratively denoise into a sample. Stable Diffusion, DALL·E 3, Midjourney. Best for: images, video, audio.
  • GANs — a generator makes fakes, a discriminator spots them; they improve together. Mostly superseded by diffusion. Best for: fast, narrow image tasks.
  • VAEs — compress data into a smooth latent space, then decode. Power the latent space inside diffusion models. Best for: representation learning / compression.
You want to generate... Use
Text, code, structured output Autoregressive transformer (LLM)
Images, video, audio Diffusion
Fast narrow images (faces, upscaling) GAN
A smooth latent space / compression VAE

The rest of this guide focuses on text — where most developers start and can ship the fastest.

3. GPT and transformers, demystified

GPT = Generative Pre-trained Transformer. Generative (produces text), Pre-trained (learned general language from a huge corpus first), Transformer (the architecture underneath). A transformer processes tokens through stacked blocks, each with self-attention + a feed-forward network (with residuals + LayerNorm), × N blocks.

Self-attention — the key idea. For each token the model computes how much to "pay attention to" every other token. Each token emits a Query (what am I looking for?), a Key (what do I offer?), and a Value (what I contribute). Attention weight = how well a query matches each key; output = weighted blend of values:

import torch, math import torch.nn.functional as F def attention(Q, K, V): # Q, K, V: (seq_len, d) scores = Q @ K.T / math.sqrt(Q.size(-1)) # relevance of each token to each other weights = F.softmax(scores, dim=-1) # turn scores into a distribution return weights @ V # weighted blend of values 

That single operation — match, weight, blend — repeated across heads and layers, is the entire reason transformers work. GPT uses only the decoder half with a causal mask so each token can only see the past — exactly what you need to write text left-to-right.

4. Language modeling & generation

A language model is trained on one task: given the previous tokens, predict the next one (cross-entropy on the shifted sequence). Astonishingly, that single objective over trillions of tokens is enough to produce models that write code, summarize, and answer questions — the capability emerges from scale. Perplexity measures it: roughly "how many tokens is the model choosing between?" (5-20 for a good in-domain model; tens of thousands for random).

Generation is a sampling loop — predict a distribution, pick one token, append, repeat until EOS. The same model produces wildly different text depending on how you sample:

def sample(logits, temperature=0.8, top_p=0.9, top_k=None): logits = logits / max(temperature, 1e-5) # <1 sharper, >1 wilder if top_k is not None: v, _ = torch.topk(logits, top_k) logits[logits < v[-1]] = float('-inf') probs = F.softmax(logits, dim=-1) # top-p (nucleus): smallest set of tokens summing to p, then renormalise and sample return torch.multinomial(probs, 1).item() 

Rules of thumb: factual/code temperature=0.0-0.3 (deterministic, fewer mistakes); chat 0.7, top_p=0.9 (coherent, not robotic); creative 0.9-1.1 (more surprising, more risk). "The model gave a weird answer" is often a decoding problem, not a model problem.

5. Ethics — the part that decides if you can ship

Generative models introduce risks classifiers never had, because they produce content rather than labeling it.

  • Hallucination — they predict plausible text, not true text; truth isn't in the loss function. Ground with RAG (answer only from provided context), cite sources, and never use raw output for high-stakes facts without review.
  • Bias — models learn their training data's biases. Test across groups; don't use them for consequential decisions about people without audits.
  • Copyright — output can reproduce training data; the legal status is unsettled. Keep a human in the loop; check code against licenses.
  • Privacy / PII — redact before the prompt leaves your network (the boundary is your responsibility); use enterprise endpoints that don't train on your inputs; don't log raw PII.
  • Misuse & cost — rate-limit and authenticate endpoints; use the smallest model that does the job; cache and batch.
import re PII = { 'email': re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"), 'phone': re.compile(r"\b(?:\+?\d{1,3}[-.\s]?)?\(?\d{3,5}\)?[-.\s]?\d{3,4}[-.\s]?\d{3,4}\b"), 'ssn': re.compile(r"\b\d{3}-\d{2}-\d{4}\b"), } def redact_pii(text: str) -> str: for label, pattern in PII.items(): text = pattern.sub(f'[REDACTED_{label.upper()}]', text) return text 

The ship checklist: PII redacted before leaving your network; high-stakes outputs grounded and/or human-reviewed; smallest model that meets quality; endpoints authenticated + rate-limited; users told they're talking to AI; no raw-PII logs; tested for obvious bias.

6. The project — a real text-generation app

A FastAPI service that generates text with streaming (SSE), PII redaction, a content guardrail, and a Gradio UI — working with either a hosted API (OpenAI/Azure) or a local open-weight model. The request flow: redact PII → content guardrail → rate limit → stream tokens back.

# app/main.py (abridged) — the safe boundary around the model @app.post('/generate') async def generate(req: GenRequest, request: Request): check_rate_limit(request.client.host) # naive in-memory; use Redis in prod if is_blocked(req.prompt): # content guardrail raise HTTPException(400, 'Prompt violates usage policy') safe_prompt = redact_pii(req.prompt) # PII never leaves the process raw async def event_stream(): async for token in stream_generate(safe_prompt, req.temperature, req.max_tokens): yield f"data: {json.dumps({'token': token})}\n\n" yield f"data: {json.dumps({'done': True})}\n\n" return StreamingResponse(event_stream(), media_type='text/event-stream') 

The backend is pluggable — LLM_BACKEND=openai streams from the OpenAI API, LLM_BACKEND=local streams from a local Hugging Face model via TextIteratorStreamer. The Gradio UI reads the SSE stream and renders tokens as they arrive. You get the three things most demos skip: streaming, PII redaction, and a safety guardrail. For production: real moderation (OpenAI moderation / Llama Guard), Redis rate limiting, auth, Langfuse observability, response caching, vLLM, and a golden eval set.

The mental model

Generative AI feels magical, but the model is simple: a model that learned the shape of data well enough to produce new samples of it, run inside a loop, shaped by your decoding choices, and made safe by the guardrails you put around it. Three habits: pick the smallest model that works (capability beyond your need is wasted cost and carbon); treat the boundary as your responsibility (redact, ground, rate-limit, disclose); and measure, don't vibe (build an eval set before tuning prompts or swapping models).

This is a condensed version. The full guide — with every section in depth, perplexity, all the decoding code, the full ethics treatment, and the complete FastAPI + Gradio project (safety, pluggable backend, run instructions) — is on PrepStack:

→ Introduction to Generative AI — Generative Models, GPT & Transformers, Language Modeling, Ethics, and a Real Text-Generation App