How to Build an Offline, On-Device Voice-to-Voice AI Agent in React Native

We’ve spent the past year helping teams like Volkswagen, Verbali and LunaFit ship voice agents that run entirely on the phone. This is what we learned building a full voice-to-voice agent in React Native, from the wake word to the spoken reply.

I’m the founder of DaVoice, and I drafted this guide with AI help, which fits the subject. Our example app ships how-to files (englishHowTo.md and AI_AGENT_INTEGRATION_GUIDE.md) written for AI coding agents. Hand them to Claude Code, Cursor or Copilot along with your codebase, and the agent can add the wake word, speech-to-text, local LLM and on-device voice to your app step by step, in the order that works. I checked every detail here against the repo.

Short answer: To build an on-device Voice-to-Voice AI Agent in React Native, chain five stages that all run on the phone: a wake word detector, optional speaker verification, speech-to-text (STT), a small local LLM that calls your app's tools, and on-device text-to-speech (TTS). Stream the LLM's reply into TTS sentence by sentence, and hand the microphone carefully between stages. This tutorial does exactly that with the open-source DaVoice React Native example app and three npm packages from the DaVoice Voice AI Agent SDK: react-native-wakeword, react-native-davoice and react-native-davoice-llm.

The result is an agent that listens for "Hey Coach", understands "open the rear right window", runs a real function in your app, and answers out loud: "I opened the rear right window." No cloud LLM, no per-token bill, and no audio leaving the device during the voice loop.

What "token-free" means here. A local LLM still processes tokens. "Token-free" means you pay no cloud LLM inference token fees, because inference runs on the user's phone. DaVoice SDKs are licensed separately; evaluation and non-commercial proofs of concept are free, and production use needs a commercial license.

Used in production

The same on-device voice stack already runs in shipped products:

• Verbali (MaTalk AI, an AAC app for non-verbal children) moved its wake word, speech-to-text and text-to-speech fully on-device with DaVoice, so no audio leaves the tablet and voice keeps working without internet. In Verbali's own write-up, DaVoice reached 97.65% wake word accuracy with zero false positives observed on their test setup, versus about 75% for the other engines they evaluated. In their words: "We have no affiliation with DaVoice beyond being a happy customer."
• Volkswagen chose DaVoice for OTTO, its generative AI car assistant launched in Brazil (customer story · video).
• LunaFit built its entire voice-to-voice AI agent on DaVoice, including speaker-aware speech-to-text that hears its user in a noisy gym (video). The interruption pattern in section 8 comes from LunaFit's workout mode.

Table of contents

1. What you will build
2. Architecture: wake word → STT → local LLM → TTS
3. What is really offline (and what is not)
4. Prerequisites
5. Run the DaVoice example app
6. Implementation, stage by stage
7. Add the agent to your existing React Native app
8. Advanced: speaker verification, voice cloning, interruptions
9. Testing and troubleshooting
10. FAQ
11. Resources

1. What you will build

A hands-free assistant inside a React Native app (iOS and Android) that:

• listens continuously for a custom wake word ("Hey Coach" in the example);
• optionally checks that the speaker is the enrolled owner;
• transcribes the request in real time;
• lets a 0.6B-parameter on-device model decide whether to call one of your app's tools, ask a clarifying question, decline, or hand the request to an (optional) remote model;
• speaks the result with an on-device neural voice, which can be a cloned voice.

The example ships two demos you can try immediately:

Honest status from the repo: the automotive model is labeled a test model, still in training. On DaVoice's frozen automotive test set it gets about 82% of Brazilian Portuguese commands exactly right (tool + arguments) and picks the right tool about 90% of the time. Every car action is simulated and checked by app-side guards. Treat it as a preview of the voice pipeline, not final accuracy.

2. Architecture: how a Voice-to-Voice AI Agent works on-device

 ┌────────────────────────── on the phone ──────────────────────────┐ microphone ──► │ Wake word ──► Speaker ──► Speech-to- ──► Local LLM │ │ "Hey Coach" verification text (STT) Agent │ │ (react-native- (optional, (react-native- (react-native-│ │ wakeword) owner only) davoice) davoice-llm) │ │ │ ▲ │ │ tool call ──►│ │result│ │ App tools (JS) │ │ │ │ speaker ◄──── │ Text-to-speech (react-native-davoice) ◄── sentence-by-sentence ─┘ │ └──────────────────────────────────────────────────────────────────┘ optional, needs network: remote model hand-over, web search, MCP 

The loop, step by step:

1. Wake word runs continuously at low power. When it fires, detection is paused (not stopped) and the mic is handed to STT.
2. Speaker verification (optional) tags every STT result with speakerId / speakerIds, and can restrict the wake word to the enrolled voice.
3. STT streams partial results. The end of the user's turn is a silence timeout (2,000 ms in the example).
4. The local agent (Agent.turn) gets the transcript plus history. The fine-tuned model decides: call a tool, ask, decline, or hand over.
5. Tool calling: your JavaScript handler runs and returns toolReply(result, sentence). The app, not the model, says what actually happened, and the turn ends without a second model round.
6. TTS speaks each sentence as soon as it is complete. STT is paused while the agent speaks, then the mic reopens.

Why a pipeline instead of one speech-to-speech model? Each stage can be tuned, tested and replaced on its own; tools can be gated by speaker identity; and a 0.6B action model stays small enough for a phone.

3. What is really offline (and what is not)

So: once models are downloaded, the core loop (wake word → STT → local agent → app tools → TTS) works in airplane mode for supported languages. Anything labeled optional above is cloud and is not part of the offline pipeline.

4. Prerequisites

• Real devices. Wake word, speaker verification, STT and the LLM need a real microphone and real memory. Use simulators for UI only.
  • iPhone: iPhone 11 or newer per DaVoice's guide (Metal GPU on A14+; CPU on A13).
  • Android: 64-bit ARM (arm64-v8a), minSdkVersion 29.
• React Native CLI app. The example uses React Native 0.82.0 and React 19.1.1. (An Expo version lives in the same repo under expo/; it needs a development build, not Expo Go.)
• Node 18+, Xcode + CocoaPods, Android Studio.
• Git LFS, because model files are stored with LFS.
• A DaVoice license key for your own app (the example contains a time-limited demo key; don't copy it). Contact info@davoice.io.
• Disk/network for the first run: the car demo downloads about 780 MB (model + pre-warmed state) once.

Versions used by the example (example/package.json) - use these together, since all three DaVoice packages ship native binaries:

"react-native": "0.82.0", "react-native-davoice": "1.0.140", "react-native-davoice-llm": "1.0.13", "react-native-fs": "^2.20.0", "react-native-wakeword": "1.1.171" 

5. Run the DaVoice example app

git clone https://github.com/frymanofer/ReactNative_DaVoice.git cd ReactNative_DaVoice git lfs pull # model files are LFS objects cd example npm install # the project uses npm / package-lock.json cd ios && pod install && cd .. npx react-native run-android # or: npx react-native run-ios 

Useful extras from the repo's how-to:

adb reverse tcp:8081 tcp:8081 # Android debug build over USB (Metro) adb logcat | grep ReactNativeJS # JS logs on Android cd android && ./gradlew :app:assembleRelease # Android release build 

First run walkthrough:

1. Pick a language (English and Brazilian Portuguese are built in; others download once).
2. Accept the microphone permission.
3. Pick a voice, or clone a new one from a recording.
4. Speaker verification: create a new voice signature (5 samples), reuse a saved one, or skip.
5. Say "Hey Coach", then pick Local AI · Car demo.
6. Wait for the loading screen (download → load into memory → warm up). The car demo ships a pre-warmed prompt state, so after the download it is ready in about a second.

Then try: "open the rear right window", "set the temperature to 22 degrees", "lock the car", "how much range do I have?". Incomplete requests should trigger a question: "open a window" → "Which window?".

A general question like "tell me a fun fact about Brazil" returns "I need to send this to a remote Model…". That is expected: no remote model is configured, and the 0.6B model is trained to act, not to chat.

6. Implementation, stage by stage

The example's voice logic is split into copyable modules under example/src/. App.tsx is the orchestration layer. The snippets below are the minimal APIs documented in the repo, plus excerpts from the real modules.

6.1 Install and native setup

npm install react-native-wakeword react-native-davoice react-native-davoice-llm react-native-fs cd ios && pod install && cd .. 

Android - each DaVoice package ships its native library as a local Maven repo. From example/android/build.gradle:

// minSdkVersion = 29 allprojects { repositories { maven { url "${project(":react-native-wakeword").projectDir}/libs" } maven { url "${project(":react-native-davoice").projectDir}/libs" } maven { url "${project(":react-native-davoice-llm").projectDir}/libs" } } } 

Add RECORD_AUDIO and INTERNET (for the one-time model download) to AndroidManifest.xml.

iOS - in Info.plist: NSMicrophoneUsageDescription, NSSpeechRecognitionUsageDescription, and UIBackgroundModes → audio if the wake word should listen in the background. For the LLM, add the entitlement the example uses:

<key>com.apple.developer.kernel.increased-memory-limit</key> <true/> 

Metro - let require() resolve model files (metro.config.js):

resolver: { assetExts: [...defaultConfig.resolver.assetExts, 'dm', 'onnx', 'wav', 'mp3'], }, 

Model files loaded by name (wake word hey_coach_model_28_22012026b.dm, speaker verification sv.dm) go into android/app/src/main/assets/ and into the Xcode target's Copy Bundle Resources.

6.2 Wake word (react-native-wakeword)

From the repo's integration guide:

import { createKeyWordRNBridgeInstance, setWakewordAudioRoutingConfig } from 'react-native-wakeword'; await setWakewordAudioRoutingConfig(defaultAudioRoutingConfig); // iOS routing, from src/wakeword/index.ts const inst = await createKeyWordRNBridgeInstance('main', false); // One or more models: names, thresholds, buffer counts, ms between callbacks. await inst.createInstanceMulti(['your_phrase_model.dm'], [0.999], [3], [1000]); const sub = inst.onKeywordDetectionEvent((phrase: string) => onWakeWord(phrase)); if (!(await inst.setKeywordDetectionLicense(DAVOICE_LICENSE_KEY))) { /* show license error */ } await inst.startKeywordDetection(0.999, true); 

In the callback, pause detection before giving the mic to STT, and un-pause afterwards:

await inst.pauseDetection(false); // false: keep the mic session // ... STT / LLM / TTS ... await inst.unPauseDetection(); 

Tuning knobs: threshold (0-1) and bufferCnt trade false triggers against misses (the example uses 0.999 / 3); msBetweenCallbacks debounces repeats. Custom wake phrases are separate models you get from DaVoice.

6.3 Speech-to-text and text-to-speech (react-native-davoice)

STT and TTS share one Speech object because they share one audio session:

import Speech from 'react-native-davoice/speech'; await Speech.setAudioRoutingConfig(defaultAudioRoutingConfig); // iOS; same object as the wake word if (!(await Speech.setLicense(DAVOICE_LICENSE_KEY))) { /* show license error */ } await Speech.initAll({ locale: 'en-US', model: require('./assets/models/model_ex2_rich_kate.dm') }); await Speech.pauseSpeechRecognition(); // start paused; un-pause when it's the user's turn Speech.onSpeechPartialResults = (e) => { /* e.value[0]: text so far */ }; Speech.onSpeechResults = (e) => { /* e.value[0]: final text; e.speakerId when verification is on */ }; Speech.onSpeechError = (e) => { /* log; most codes are recoverable */ }; await Speech.unPauseSpeechRecognition(-1); // listen await Speech.pauseSpeechRecognition(); // stop listening (e.g. before TTS) await Speech.speak('I opened the window.', 0, 1.0); // text, speaker index, speed 

initAll can take many seconds the first time, so the example wraps it in a timeout and keeps the wake word running if it stalls (initializeWakewordBootstrap in src/wakeword/index.ts):

await withTimeout( initializeSpeechLibrary(enrollmentJsonPath ?? null), speechInitTimeoutMs, 'Speech.initAll', ); 

6.4 The local LLM and the Agent (react-native-davoice-llm)

The package has two layers: an engine (setLicense, loadModel, generateStream, prefill, cancel, saveState, loadState) and an Agent that runs a whole assistant turn - prompt assembly, tool calls, guards, memory and streaming - configured by an agent pack of JSON files instead of code. DaVoice's on-device LLM guide covers model choice in more depth.

Models. The GGUF files are downloaded at runtime and checked by sha256 - never bundled in the binary. From src/local_llm/constants.ts:

{ id: LOCAL_LLM_DAVOICE_AUTOMOTIVE_R4B, displayName: 'DaVoice Automotive Agent 0.6B', filename: 'davoice-automotive-qwen3-0.6b-r4b-it100-q4_0.gguf', url: 'https://pub-27f94b1602c647ab95a601086cf52820.r2.dev/agent-llm/auto5_r4b_it100-q4_0.gguf', sizeBytes: 381_565_920, sha256: '1c078e961c5f4e16ae0f667d213fa06de53f3f4bc92c9248427bdb37237bd205', contextSize: 8192, chatTemplate: CHAT_TEMPLATE_PRESETS.qwen3.template, // ... } 

The picker also lists general models (Qwen2.5 1.5B/3B with chatml, Gemma 3n E2B/E4B with gemma). They are larger and slower; the 0.6B agents are the defaults.

Minimal agent setup (from the repo's englishHowTo.md):

import { setLicense, loadModel, Agent, toolReply } from 'react-native-davoice-llm'; await setLicense(LICENSE); await loadModel(pathToGguf, { contextSize: 8192 }); const agent = await Agent.create({ pack: { agent: require('./agent_automotive/agent.json'), skills: [] }, databasePath: `${RNFS.DocumentDirectoryPath}/davoice_agent_car.db`, chatTemplate: 'qwen3', }, { onRequest }); await agent.configure({ intentRouter: false, webSearchTool: true, timeTool: true, guards: { /* see provider.ts */ } }); await agent.setTools(myTools); // a handler returns toolReply(result, 'I opened the window.') to say it itself const r = await agent.turn({ text, history }, (ev) => { if (ev.type === 'text') speak(ev.text); }); // agent.cancel() stops a running turn 

The example's real configuration for the fine-tuned models (src/aichat/provider.ts, condensed - the original reads temperature and memory options from the model entry) shows the settings that matter:

agent.configure({ chatTemplate: model.chatTemplate, remote: !handsOver ? false : LOCAL_LLM_LAYER2 === 'device' ? { enabled: true, answer: 'layer2' } : { enabled: true, timeoutMs: 15000 }, intentRouter: model.usesRouter !== false, // false: the model decides with every tool webSearchTool: model.modelTools === true, timeTool: model.modelTools === true, memory: { profile: 'small' }, sampling: { temperature: 0 }, // greedy for tool calling guards: callGuardSettings(model.callGuards ?? 'off', model.historyAsRun === true), historyAsRun: model.historyAsRun === true, guidedToolCalls: model.guidedToolCalls === true, promptOrder: model.promptOrder ?? 'trained', }); 

Create the agent once and answer its requests (src/agent/LocalAgent.ts). onRequest is where the app decides about remote hand-over and sensitive tools:

onRequest: async (request) => { if (request.type === 'remote') return answerRemoteRequest(request as AgentRemoteRequest); if (request.type === 'authorizeTool') return { authorized: isToolAuthorized(request as AgentApprovalRequest) }; if (request.type === 'confirmTool') return { confirmed: await askToConfirm(request as AgentApprovalRequest) }; throw new Error(`The app does not answer "${request.type}" requests.`); }, 

6.5 Tool calling: define tools in MCP shape, let the app confirm

Tools are plain JSON in MCP shape (name, description, inputSchema, optional execution). From src/car/carToolDefinitions.h3.json:

{ "name": "set_window", "description": "Opens or closes windows, fully, halfway or just a crack.", "inputSchema": { "type": "object", "properties": { "window": { "type": "string", "enum": ["all", "front_left", "front_right", "rear_left", "rear_right", "driver", "passenger", "front", "rear", "left", "right"], "description": "Which window: ... Ask if the user did not say which" }, "position": { "type": "string", "enum": ["open", "close", "half", "crack"], "description": "open (lower it), close (raise it), half, crack (a little)" } }, "required": ["window", "position"] } } 

A sensitive tool declares an authorization scope, and the agent core never runs it without a "yes" from the app:

"execution": { "authorization": { "scope": "door_locks", "message": "Unlock the car?" } } 

The JS handler does the work and returns the sentence to speak. This is the pattern in src/car/carTools.ts (trimmed):

import { toolReply } from 'react-native-davoice-llm'; function toTool(t: any): ToolDefinition { return { name: t.name, description: t.description, inputSchema: t.inputSchema, ...(t.execution ? { execution: t.execution } : {}), handler: (rawArgs: Args) => { const args = rawArgs ?? {}; const run = HANDLERS[t.name]; try { if (!run) throw new Refused("Sorry, this car doesn't have that function."); const { summary, result, say } = run(args); // The app says what it did and the turn ends there: no second model round. if (say) return toolReply({ ok: true, ...result }, say); return { ok: true, ...result }; } catch (error) { const reason = error instanceof Error ? error.message : String(error); if (error instanceof Refused) return toolReply({ ok: false, refused: reason }, reason); throw new Error(`Not done: ${reason}`); } }, }; } 

Why toolReply? Two reasons from the repo's field testing: the spoken confirmation is built from the real result ("The front right door is already locked....