Running Large Language Models (LLMs) locally on a smartphone used to feel like science fiction. But with quantized GGUF architectures and optimized native runtimes like llama.rn, modern mobile chipsets can now execute capable models (like Llama 3.2 1B/3B and Qwen 2.5 1.5B) entirely on-device.

No cloud dependencies, zero monthly OpenAI/Anthropic API bills, and 100% data privacy for users.

However, moving from a proof-of-concept script to a real, production-ready React Native app reveals three major engineering hurdles:

  1. Thermal runaway & OS OOM kills: Continuous token generation pushes the CPU/GPU hard, causing the Android OS Low-Memory Killer (OOM) to terminate the app.
  2. Background thread freezing: The moment a user minimizes the app or locks their screen, the JS thread pauses and streaming stops.
  3. Document and image parsing: Processing PDFs, DOCX, and images on-device without calling third-party cloud APIs.

Here is how I tackled each of these challenges in Expo 52.

  1. Adaptive C++ Duty-Cycle Throttling

When running native inference via llama.rn, the decode loop consumes 100% of available processor cores by default. On a desktop or server with active cooling, this is fine. On a smartphone, the chassis rapidly heats up, triggering hardware thermal throttling and aggressive OS process pruning.

The Solution:

We patched the native C++ decode loop to introduce configurable, adaptive duty-cycle pauses between generation slices. By yielding execution for a few milliseconds between token chunks:

  • SoC temperature drops by over 8°C.
  • Sustained memory overhead remains predictable (850MB – 1.3GB RAM).
  • The Android Low-Memory Killer never triggers.
  1. Dedicated Android Foreground Service

Mobile operating systems are ruthless with background tasks. If generation takes 15–20 seconds and the user locks their screen to put their phone in their pocket, the generation usually dies.

The Solution:

We built a custom native Android Foreground Service with wake locks and persistent notification controls. While inference runs:

  • A notification bar shows live streaming progress with a one-tap "Stop" action.
  • The CPU wake lock guarantees generation finishes cleanly even if the phone screen is turned off.
  1. Multimodal OCR & Local Document Extraction

A modern AI assistant needs to understand documents and photos. Rather than shipping user documents to a cloud endpoint:

  • Camera & Photo OCR: We integrated expo-mlkit-ocr to extract text from images directly using Google ML Kit on-device.
  • Native Document Reader: A custom Android native module extracts text from local PDFs (up to 64 pages) and Word .docx files on the device filesystem.
  1. Atomic SQLite Response Checkpointing

If a user receives a phone call during token generation, the app might be paused. To prevent losing generated text, we stream tokens into a local SQLite database every 2 seconds. If anything interrupts generation, the conversation state is preserved and seamlessly resumable.

Real Hardware Benchmarks

Tested on a physical Motorola Edge 50 Neo (MediaTek Dimensity 7300, 8GB RAM, Android 14/15):

  • Llama 3.2 1B Instruct (Q4_K_M): ~18–25 tokens/sec
  • Qwen 2.5 1.5B Instruct (Q4_K_M): ~10–14 tokens/sec
  • Memory footprint: 800MB–1.4GB RAM ceiling
  • Cold model load time: ~1.2s from flash storage

Explore the Architecture & Starter Kit

I've made the architecture breakdown, benchmark documentation, and screenshots publicly available on GitHub:

👉 GitHub Repository: https://github.com/Itsvks17/localai-mobile

For developers and indie founders looking to build their own local AI mobile apps without spending weeks debugging native C++ throttles and Android foreground services, I also packaged a turnkey starter kit with all custom native modules and 119 automated regression tests on Gumroad:

👉 Starter Kit on Gumroad: https://darkryoshin.gumroad.com/l/localai-mobile

Feel free to star the repo or ask questions below about mobile GGUF quantization, Expo config plugins, or memory tuning!