An AI health companion that answers — and places — phone calls in Hindi and English, built on LiveKit and Murf Falcon over ten days of shipping one feature at a time. By Siddharth Agarwal · Health Access track, 10-day voice-agent build · github.com/siddharthagarwal1983/sid-voice-for-bharat The problem, and who it's for Most healthcare-access tools assume a smartphone, a data plan, comfortable reading, and one language at a time. A phone call assumes none of that — everyone already knows how to make one. India's front line of health access usually isn't a hospital — it's a call to an ASHA worker, a PHC helpline, or the 108 ambulance line. The caller may be switching between Hindi and English mid-sentence, may not read comfortably, and just wants to know one thing: is this serious, and where do I go? I built HealthMitra for the Health Access track of a 10-day voice-agent challenge, aimed at three kinds of caller: Someone trying to decide whether a symptom needs home care, a Primary Health Centre visit, or an ambulance, right now, without waiting on hold. Someone asking whether they qualify for a government scheme like PM-JAY (Ayushman Bharat) and what documents they'd need to bring. An ASHA community health worker who needs a fast reference answer mid-visit, not a portal to log into. Voice meets all three where they are — zero learning curve, works on any phone, and in whichever language they're already speaking. What HealthMitra actually does HealthMitra opens every call in English and re-checks the caller's language on every single turn — the moment they switch to Hindi it replies in Hindi, in proper Devanagari script, and switches straight back the instant they speak English again, without ever commenting on the switch. From there, a call can go several places: A described symptom gets run through a deterministic triage engine — not the model's own judgement — which returns one of three routes: home_care, phc visit, or emergency → call 108. A scheme-eligibility question is answered only from a real knowledge base, never from the model's memory of how PM-JAY "probably" works. A caller who wants to book, check, or cancel a clinic visit, an imaging scan, or a lab test is handed off mid-call to one of three specialist agents built for exactly that. Something the assistant can't resolve — a genuinely unanswered question, or a caller who explicitly wants a human — gets logged as an escalation, with sensitive details stripped before anything is saved. Some calls are outbound: HealthMitra dials out for a medication reminder, a vaccination reminder, or a check-in after a previous triage or escalation, and knows to leave a short message and hang up if it reaches voicemail instead of a person. Every call — inbound or outbound — feeds a small analytics dashboard that tracks outcomes, failure reasons, and response latency, which is how most of the numbers later in this post were measured. How the system works LiveKit is the transport layer — it moves raw audio between the caller and the agent over a WebRTC room (and bridges to the phone network over SIP for outbound calls). Inside that room, four stages turn sound into a response and back into sound: Caller (phone / browser) │ audio in ▼ Deepgram Nova-3 — multi-language STT → text + detected language │ ▼ Gemini 3.5 Flash-Lite — reasoning + tool calls │ ▼ Murf Falcon TTS — voice: Namrita → audio out │ ▼ Caller hears the reply (~190ms to first audio) Side systems: SQLite (caller memory, appointments, escalations, analytics) · Escalation webhook (PII-redacted, notifies a human queue) · Knowledge base (full-text search over scheme documents) All four stages run inside one LiveKit room — STT and TTS run over the same room as the caller's own audio track. Why the TTS leg is the one a caller actually feels Of the four stages, TTS latency is the one with nowhere to hide. On a web chat you can show a typing indicator; on a phone call there's only the audio channel — any lag between the caller finishing a sentence and the reply starting reads as dead air, or as the line having dropped. Murf Falcon publishes a ~130ms time-to-first-audio benchmark; across my own test calls the agent averaged 191ms to first audio, measured the same way whether the caller was in the browser demo or on an outbound phone call over SIP. That number matters more for the phone leg than the web one — a browser caller has a visible "Listening" state to reassure them something is happening, but a phone caller has no screen at all, so the TTS engine's own speed is the entire experience of the assistant feeling responsive rather than stuck. The features that tell the story 1. An Indian voice, tuned for code-mixing — The agent speaks as Namrita, one of Murf Falcon's Indian voices, over a Deepgram speech recognizer running in a multi-language mode that tags which language each utterance was actually in, rather than assuming one for the whole call. The assistant tracks the caller's current language as a live fact, re-checked on every turn, and always writes Hindi in Devanagari script (देवनागरी लिपि) — never Latin-alphabet "Hinglish" — because the TTS engine mispronounces transliterated Hindi. 2. A clear identity, and hard refusals — The prompt gives HealthMitra an identity, objectives, and knowledge boundary before anything else: it triages and routes, it does not diagnose, and it will never name a specific medicine. If a caller asks what to take, it deflects to "please take what your doctor prescribed, or visit your nearest PHC" in whichever language they're speaking — and if a red-flag symptom comes up (chest pain, severe breathlessness, sudden weakness, heavy bleeding, an infant fever), it interrupts the normal flow immediately with a scripted line pointing to the 108 ambulance service, before anything else happens. 3. Memory that asks first — Nothing about a caller is preloaded. A caller profile — age band, ongoing conditions, district, last triage outcome — is only written after the agent explicitly asks "may I remember this for next time?" and the caller agrees; fields are short structured tags, never a free-text medical note. Callers can also ask to be forgotten entirely, which deletes the row after one confirmation. 4. Tools grounded in real data, not the model's memory — Symptom triage runs through a hand-written, deterministic rule set that always returns one of three routes — emergency, PHC visit, or home care. The model narrates the result in its own words but doesn't decide the route itself, so the same symptom routes the same way every time. Scheme-eligibility questions go through a full-text search over real reference documents, and the agent is instructed to say "I'm not sure" rather than answer from its own training data. 5. Escalation to a human, safely — When the agent hits something it can't resolve, it tells the caller exactly what it wants to send to a human and asks permission before creating the ticket. The summary is redacted before it's stored or sent to a webhook — no OTPs, PINs, or account numbers, and no raw transcript, just a qualitative description and an urgency level. The agent is also told never to promise a callback time it can't guarantee — only that a human will review it. 6. Handoffs to specialist agents — Booking, checking, rescheduling or cancelling an appointment is handled by one of three narrow specialists — clinic/OPD, radiology/imaging, or pathology/lab — each with exactly one job and no tools outside it. They share a base for the plumbing (handoff bookkeeping, facility lookup, appointment CRUD) and hand back to the main assistant the instant the caller mentions anything outside their scope, like a symptom or an emergency. 7. Outbound calls — Not every call is inbound. A small outbound-calling script can dial a caller for a medication reminder, a vaccination reminder, or a follow-up after a previous triage or escalation. Outbound calls open differently — the agent identifies itself and the reason immediately, since the person didn't choose to pick up — and it's trained to recognize a voicemail greeting and leave one short message instead of talking to an answering machine. 8. A call-analytics dashboard — A single Python http.server script reads straight from the SQLite database and renders stat cards, an outcome bar chart, a daily-volume chart, and a filterable table of recent calls — no dashboarding framework, just stdlib HTML generation. From my own test conversations so far (dev-testing calls, not real patient traffic): Metric Value Call sessions 63 Recorded successes 13 / 25 Avg. time-to-first-audio 191ms Saved caller profiles 9 Automated tests passing 13 / 13 Time-to-first-audio is measured as Murf Falcon's TTS time-to-first-byte, sampled from LiveKit's metrics events — the thing a caller actually experiences as "how long until it started talking." Challenges: three bugs that taught me how the framework actually behaves Every one of these looked correct by reading the prompt or the code. Each was only findable by pulling the real call transcript out of SQLite and cross-referencing the backend's tool-call logs — none of them showed up until I stopped guessing and started reading the evidence. Incident 01 — The Hindi that wouldn't let go Symptom: Mid-handoff introduction lines came out in Hindi during an otherwise all-English call. Cause: A one-off instruction ("say this in English: ... / in Hindi: ...") gave the model a literal bilingual template to copy from, and it sometimes just parroted the Hindi half regardless of what the caller was actually speaking. Fix: Pass the caller's current language in as a computed fact, not a template to choose from: "The caller is CURRENTLY SPEAKING ENGLISH — respond IN ENGLISH." A directive generalizes; an example pair gets copied verbatim, especially on a fast, cheap production model. Incident 02 — The specialist that couldn't let go Symptom: A caller who'd gotten mild-headache advice earlier in the call, then asked to book a lab test, got bounced back and forth between the main assistant and the pathology specialist indefinitely — until they gave up and hung up. Cause: Specialists inherit the full conversation history so the caller never has to repeat themselves. But the specialist's own "hand back if this drifts outside my scope" guardrail was matching the earlier, already-resolved symptom mention in that history, not the live request. Fix: Made it explicit in each specialist's prompt that history from before the handoff was already handled and is not itself a reason to hand back — only a new thing the caller says to the specialist counts. Added a logging-only counter that flags 3+ handoffs without an intervening real user turn, as a tripwire if it happens again. Incident 03 — The call that went quiet Symptom: Right after a handoff, the call would sometimes just go silent for 15–25 seconds until the caller spoke again — indistinguishable, from the caller's side, from the app crashing. Cause: The tool told the agent to "tell the caller you're connecting them, then call the transfer tool" in one turn. That's two instructions, not one — the production model would sometimes speak the narration and skip the tool call. A second bug compounded it: the handoff was pairing a spoken message together with the new agent instead of switching cleanly, which forced an extra reply turn from the old agent that raced the new specialist's own introduction. Fix: Never ask the model to narrate a handoff — call the tool immediately, and let the new agent own its own introduction line the moment it takes over, deterministically, instead of leaving it to the model to freelance. Here's the handoff tool after the fix: # Call this tool immediately, in the same turn, once you decide the # caller needs this specialist — the specialist introduces itself # automatically once it takes over, so don't narrate the handoff first. @function_tool async def transfer_to_clinic_specialist( self, context: RunContext, reason: str ) -> Agent | str: # bare Agent on success — never (message, Agent) return await self._start_appointment_specialist_handoff( ClinicAppointmentSpecialist, reason ) Build your own: a practical start Any voice agent — a health line, a receptionist, a language tutor — is the same four pieces wired together: Real-time transport (LiveKit) — moves raw audio both ways over WebRTC, and optionally bridges to the phone network over SIP so the same agent can take real calls. Speech-to-text (Deepgram) — turns the caller's audio into text, streaming, as they talk. An LLM (Gemini, or any LiveKit-supported model) — decides what to say and which tools to call. Text-to-speech (Murf Falcon) — turns the reply back into audio and streams it back through the same room. 1. Clone and install — Start from the open murf-livekit-starter template (what this project is built on), then set up each side on its own: cd backend && uv sync cd frontend && pnpm install 2. Add your keys — never to the repo — Copy backend/.env.example → backend/.env.local, and frontend/.env.example → frontend/.env.local. Both copies are gitignored by default. Fill in your LiveKit, Murf, Deepgram, and LLM API keys there — never commit them, and never paste them into the system prompt. 3. Download models once — Run this once, from the backend folder, to pull the local turn-detector and voice-activity models the pipeline needs: uv run python src/agent.py download-files 4. Run everything — One script, from the repo root, starts LiveKit, the backend agent, and the frontend together: ./start_app.sh Then open http://localhost:3000. 5. Have a conversation — Click Start talking in the browser and speak — or skip the frontend with a terminal-only console mode for a quick text-free test loop. The repo's own test suite runs real multi-turn conversations against the actual LLM and has a second model judge the responses, instead of mocking anything. Where your prompt lives: the entire personality, objectives, and guardrails are one string — SYSTEM_PROMPT near the top of backend/src/agent.py. Change that to change what the agent does; nothing else needs to move. Troubleshooting, if you're following along The agent never answers, or the frontend says no agent is available — Check that the agent name configured on the frontend matches the one registered in your backend agent code, and that the backend process is actually running and connected to the same LiveKit project. Hindi comes out as Latin-alphabet "Hinglish" instead of sounding right — Most TTS engines mispronounce transliterated Hindi. Require the model to write Hindi in Devanagari script explicitly in the prompt — don't leave it to infer that from context. A multi-agent handoff goes silent for several seconds — Check your handoff hands off cleanly to the new agent instead of pairing it with a spoken message, and that nothing asks the model to narrate the handoff before it calls the tool — see Incident 03 above. The agent's language "sticks" to whatever it used last, even after the caller switches back — Track the caller's current language as a fact you update from your STT's per-utterance detection, and inject it as a live directive each turn — not as a bilingual example pair for the model to pick from. What I'd improve next A real circuit breaker for handoffs — the framework already exposes a way to cancel a handoff mid-flight; right now there's only a logging counter that flags loops after the fact. Live-testing the other Indian languages the prompt already allows (Marathi, Bengali) as thoroughly as Hindi has been — right now only Hindi and English have real call evidence behind them. Replacing the cached/fallback facility list with a live government facility API. Moving caller data, appointments, and escalations off local SQLite before any real deployment. A proper UI for the human escalation queue, instead of a script run by hand. Code and demos The full project — prompt, tools, specialists, dashboard, and the 13-test suite mentioned above — is public. No API keys, phone numbers, or caller data are in the repository; the numbers in this post came from local test conversations only. 🔗 github.com/siddharthagarwal1983/sid-voice-for-bharat HealthMitra · Health Access track · Built with LiveKit Agents, Deepgram, Gemini & Murf Falcon.