1. The Core Problem: The Healthcare Data Crisis & The Fragility of Naive LLMs
In modern digital health, over 80% of actionable clinical information remains trapped in unstructured formats: handwritten physician notes, scanned lab PDFs, discharge summaries, and conversational patient complaints.
When organizations attempt to solve this by deploying standard, out-of-the-box Generative AI or basic RAG (Retrieval-Augmented Generation) pipelines, they hit critical production roadblocks:
- The Medical Hallucination & Citation Problem: In clinical environments, a fabricated lab value or an unsubstantiated dosage recommendation is a severe patient safety hazard. Generic LLMs lack deterministic data grounding and verifiable chunk provenance.
- The "Document Blindness" of OCR: Medical lab reports are rarely simple text. They consist of multi-column tables with reference ranges, flags (
HIGH,LOW,CRITICAL), and abbreviations. Standard OCR fails to preserve the relational context between a test analyte, its measured value, and its reference interval. - The Semantic Disconnect (Patient Jargon vs. Clinical Terminology): A patient says, "I have a splitting headache on the left side of my temple for 3 days." Electronic Health Record (EHR) systems, however, require structured terminology such as SNOMED CT Concept ID 21522000 or ICD-10. Naive chat systems fail to extract qualifiers (laterality, severity, duration) and bind them to standard ontologies.
- Multi-Jurisdiction Knowledge Silos: Clinical recommendations differ by country. A patient in the UK operates under NHS guidelines, Ireland under HSE protocols, and the US under FDA/HIPAA rules. A monolithic LLM prompt cannot dynamically partition institutional protocols from private patient records.
- Real-Time Voice Latency & Cognitive Load: Sick or elderly patients struggle to type lengthy symptom histories. Medical systems require sub-second Voice Activity Detection (VAD), reliable Speech-to-Text (STT), and empathic, natural Text-to-Speech (TTS).
2. High-Level Architectural Solution
To resolve these architectural challenges, this platform combines a Serverless Event-Driven Microservices Architecture (AWS Lambda on Node.js 22) with a coordinated suite of AWS Specialized AI & Generative AI Services:
flowchart TD subgraph Client_Layer ["Client & Ingestion Layer"] User["Patient / Clinician"] PDF["Lab Reports / Consultant Letters (PDF)"] Voice["Mic Voice Input (PCM Audio)"] end subgraph Audio_Processing ["Audio & Speech Pipeline"] VAD["Browser VAD (Voice Activity Detection)"] Transcribe["AWS Transcribe (Speech-to-Text / en-GB)"] Polly["AWS Polly (Neural TTS / Amy Voice)"] end subgraph Document_Pipeline ["Document OCR & Ingestion Pipeline"] S3["AWS S3 (Raw Inbound Storage)"] Textract["AWS Textract (Table & Form Analysis)"] HTML["Structured HTML & Clean Markdown"] BedrockKB_Sync["Bedrock Knowledge Base Ingestion Job"] end subgraph AI_Reasoning_Engine ["AWS Bedrock Generative AI Core"] Router["Unified Chat Router Lambda"] Titan["Amazon Titan Embeddings (amazon.titan-embed-text-v1)"] ClaudeSonnet["Claude 3.5 Sonnet (Clinical Reasoning)"] ClaudeHaiku["Claude 3 Haiku (Entity Extraction & Classification)"] BedrockKB["AWS Bedrock Knowledge Bases (Dual-Store RAG)"] end subgraph Persistence_Terminology ["Data & Ontological Grounding"] CTS["SNOMED CT Terminology Server (CTS API)"] Supabase["Supabase PostgreSQL + pgvector"] end User --> Voice --> VAD --> Transcribe --> Router User --> PDF --> S3 --> Textract --> HTML --> BedrockKB_Sync --> BedrockKB Router --> BedrockKB Router --> ClaudeSonnet ClaudeSonnet --> Polly --> User ClaudeHaiku --> CTS --> Supabase 3. Deep-Dive Technical Implementation
A. AWS Textract: Preserving Relational Context in Clinical Lab PDFs
The Technical Challenge:
Lab reports contain non-linear, multi-column tables. If a patient’s Thyroid Stimulating Hormone (TSH) is 6.2 mIU/L (Reference: 0.4 - 4.0), naive line-by-line OCR extracts:
TSH 0.4 - 4.0 6.2 High The spatial relationship between the analyte, the result, and the reference range is lost, causing downstream RAG embeddings to scramble the data.
The Technical Solution:
In our ingestion pipeline (clinical-document-processor.mjs), we invoke Amazon Textract using two-phase asynchronous/synchronous analysis with feature types TABLES and FORMS:
import { TextractClient, StartDocumentAnalysisCommand, GetDocumentAnalysisCommand } from "@aws-sdk/client-textract"; const textract = new TextractClient({ region: process.env.AWS_REGION }); // 1. Kick off table and key-value analysis const startCmd = new StartDocumentAnalysisCommand({ DocumentLocation: { S3Object: { Bucket: s3Bucket, Name: s3Key } }, FeatureTypes: ["TABLES", "FORMS"] }); const { JobId } = await textract.send(startCmd); Graph Block Reconstruction:
Textract returns a directed acyclic graph of Block objects (PAGE, LINE, WORD, TABLE, CELL, KEY_VALUE_SET). Our pipeline builds an index map of all blocks by Id and reconstructs clean HTML tables (lib/textract-to-html.mjs):
- Cells are matched to row and column indices (
RowIndex,ColumnIndex). - Children block IDs (
WORDandSELECTION_ELEMENT) are dereferenced to assemble the cell text. - Relational Markdown/HTML tables are generated:
| Test Analyte | Result | Flag | Reference Range | Units | |---|---|---|---|---| | TSH | 6.2 | HIGH | 0.4 - 4.0 | mIU/L | This structured format guarantees that vector embeddings retain semantic integrity.
B. AWS Bedrock Knowledge Bases & Titan Embeddings: Deterministic RAG with Attribution
The Technical Challenge:
Medical retrieval requires both high semantic recall and zero false attributions. If a patient asks: "What did my endocrinologist recommend regarding metformin?", the system must retrieve the exact consultant letter, produce a verified similarity score, and cite the document name.
The Technical Solution:
We leverage Amazon Bedrock Knowledge Bases combined with amazon.titan-embed-text-v1. In personal-kb-handler.mjs, we execute RetrieveAndGenerateCommand with dynamic retrieval configurations:
import { BedrockAgentRuntimeClient, RetrieveAndGenerateCommand } from "@aws-sdk/client-bedrock-agent-runtime"; const bedrockAgentClient = new BedrockAgentRuntimeClient({ region: "us-east-1" }); const response = await bedrockAgentClient.send( new RetrieveAndGenerateCommand({ input: { text: userQuery }, retrieveAndGenerateConfiguration: { type: "KNOWLEDGE_BASE", knowledgeBaseConfiguration: { knowledgeBaseId: process.env.KNOWLEDGE_BASE_ID, modelArn: `arn:aws:bedrock:us-east-1::foundation-model/${process.env.BEDROCK_MODEL_ID}`, retrievalConfiguration: { vectorSearchConfiguration: { numberOfResults: 20 } }, generationConfiguration: { promptTemplate: { textPromptTemplate: CLINICAL_SAFETY_PROMPT_TEMPLATE } } } } }) ); Chunk Provenance & Score Aggregation:
Instead of blindly returning the generated string, our pipeline parses the raw citations metadata:
- We compute aggregate similarity scores across returned passages.
- We resolve S3 storage keys into time-limited presigned URLs (via
@aws-sdk/s3-request-presigner), giving the patient a one-click audit trail directly to their source PDF.
C. AWS Bedrock (Claude 3.5 Sonnet & Haiku): Clinical Reasoning & Entity Extraction
Model Tiering Strategy:
To optimize both cost and inference latency, we split our conversational pipeline:
- Tier 1 (Claude 3.5 Sonnet via Bedrock
ConverseCommand): Handles multi-turn conversational reasoning, synthesizing patient EHR history with organizational guidelines. - Tier 2 (Claude 3 Haiku): Functions as a high-speed entity extraction engine for structured data parsing.
Clinical Extraction Pipeline:
In symptom-coding.js, patient text is fed into Claude with a strict JSON schema enforcement prompt:
const EXTRACTION_SYSTEM = `You extract symptom entities from patient text. Return JSON only: { "symptoms": [ { "symptom_text_raw": string, "clinical_term": string, "severity": "mild" | "moderate" | "severe" | null, "laterality": "left" | "right" | "bilateral" | null, "body_site": string | null, "duration": string | null, "onset_context": string | null } ] } CRITICAL: clinical_term must remain symptom language (e.g., headache, sore throat). Do NOT output diagnoses.`; Ontological Binding with SNOMED CT:
Once extracted:
- The
clinical_termis queried against a local SNOMED CT Terminology Server (CTS). - If exact matching fails, scored candidate concepts are retrieved.
- Bedrock performs semantic disambiguation, selecting the most accurate SNOMED CT Concept ID (e.g., Concept
25064002for Headache), storing confidence scores, laterality, and anatomical site directly in PostgreSQL.
D. AWS Polly & AWS Transcribe: Natural, Sub-Second Voice AI
To make the system hands-free, we built an end-to-end voice loop:
[Mic Stream] ──► Web VAD ──► Base64 PCM ──► AWS Transcribe ──► Bedrock LLM ──► AWS Polly (TTS) ──► Audio Output - Client-Side VAD (
@ricky0123/vad-web): Runs an ONNX speech detection model in the user's browser. It detects the exact millisecond speech begins and stops, buffering 16kHz PCM audio without streaming dead silence. - Speech-to-Text (AWS Transcribe): Converts buffered PCM audio chunks to text (
lib/transcribe-pcm.mjs) using UK English (en-GB) medical acoustics. - Text-to-Speech (AWS Polly): In
lib/polly-synthesize.mjs, when Bedrock finishes generating its answer, plain-text sanitization runs (plainChatAnswerText) to strip markdown headers and bullet points. The sanitized text is synthesized via Polly using the Neural Voice Engine (Amy, en-GB):
import { PollyClient, SynthesizeSpeechCommand } from "@aws-sdk/client-polly"; const polly = new PollyClient({ region: process.env.AWS_REGION }); const command = new SynthesizeSpeechCommand({ Engine: "neural", VoiceId: "Amy", OutputFormat: "mp3", TextType: "text", Text: sanitizedSpokenText }); const { AudioStream } = await polly.send(command); The resulting MP3 stream is converted to base64 and played back in the browser, creating an immediate conversational bedside assistant.
E. Financial Governance & Token Audit Logging
Running foundation models in production without real-time observability is financially risky. In bedrock-invocation-audit-log.mjs, every call to Bedrock intercepts token consumption:
$$\text{Total Cost} = \left(\frac{\text{Input Tokens}}{1,000} \times P_{\text{in}}\right) + \left(\frac{\text{Output Tokens}}{1,000} \times P_{\text{out}}\right)$$
We capture:
- Exact Model ARN / Inference Profile ID
- Input & Output token counts
- Latency in milliseconds
- Calculated USD cost per request
- Stored into the
bedrock_invocation_audit_logtable in Supabase PostgreSQL for automated daily billing alarms.
4. Key Takeaways & Lessons for Production AI Engineering
- OCR Must Be Structurally Aware: Standard optical character recognition is inadequate for medical documents. Converting raw Textract blocks into hierarchical tables is essential for meaningful embeddings.
- Dual-Store RAG Outperforms Monolithic Vector Stores: Decoupling patient-specific health records (ephemeral, highly private) from institutional clinical guidelines (static, multi-tenant) prevents data leakage and ensures jurisdiction-specific compliance.
- Grounding Free Text in Medical Ontologies: Generative AI should not be the system of record. By utilizing Bedrock to extract qualifiers and binding them to SNOMED CT Concept IDs, the platform bridges conversational AI with standard hospital EHR systems.
- End-to-End Multimodality Requires Low Latency: Combining in-browser Voice Activity Detection with AWS Transcribe and AWS Polly enables sick patients to interact effortlessly through speech.
Summary Table: AWS AI Service Stack
| Service | Primary Role in Platform | Key Technical Artifact / API |
|---|---|---|
| AWS Bedrock (Claude 3.5 Sonnet) | Clinical synthesis, reasoning, safety guardrails | ConverseCommand, inference profiles |
| AWS Bedrock (Claude 3 Haiku) | Fast, cost-efficient clinical entity extraction | Few-shot JSON schema prompting |
| AWS Bedrock Knowledge Bases | Managed vector search, citation attribution | RetrieveAndGenerateCommand |
| Amazon Titan Embeddings | High-dimensional text representation | amazon.titan-embed-text-v1 |
| AWS Textract | Structural table & form extraction from medical PDFs | StartDocumentAnalysisCommand (TABLES, FORMS) |
| AWS Polly | Neural text-to-speech for conversational answers | SynthesizeSpeechCommand (Engine: neural, Amy) |
| AWS Transcribe | Speech-to-text transcription of patient symptoms | PCM streaming, StartStreamTranscription |
| AWS Comprehend Medical | PHI detection, entity anonymization (optional) | DetectPHICommand, DetectEntitiesV2Command |
