Wiring CoreML to a Quantized On-Device LLM for Streaming Text Generation

--- title: "Wiring CoreML to a Quantized On-Device LLM: KV-Cache, ANE Scheduling, and the Token Budget for iPhone 16" published: true description: "CoreML's ML Program format for on-device transformer inference — ANE scheduling, 4-bit quantization tradeoffs, and the KV-cache formula that keeps you under iOS memory limits." tags: ios, mobile, architecture, api canonical_url: https://mvpfactory.co/blog/coreml-on-device-llm-kv-cache-ane-iphone16 --- 

What We Are Building

By the end of this tutorial you will understand — and be able to implement — a streaming token generation pipeline on iPhone using CoreML. Specifically: how to schedule transformer layers across the ANE and GPU correctly, which quantization format keeps your model weights under the 2GB threshold, and how to calculate the KV-cache ceiling before jetsam kills your process in production.

Get any one of these three wrong and you are debugging OOM crashes on a user's device. Let me show you the pattern I use in every on-device inference project.

Prerequisites

• Xcode 15+ with a physical device running iOS 17 (simulator will not reflect ANE behaviour)
• coremltools 7.x installed in your Python environment
• A base transformer model checkpoint (3B parameters or smaller for iPhone 16)
• Familiarity with CoreML's .mlpackage format

Step 1 — Understand the Hardware Scheduling Problem

Apple's unified memory architecture means the CPU, GPU, and Neural Engine share the same physical memory pool. No PCIe bottleneck — but everything competes for the same budget, and iOS reclaims it without mercy.

CoreML's ML Program format (.mlpackage with a model.mlmodel spec) is your abstraction layer between Swift and that hardware. The compiler partitions your compute graph across available units. Here is where most teams go wrong: they assume the ANE and GPU handle attention and feed-forward layers identically. They do not.

The ANE is optimized for fixed-shape, low-precision matrix multiply — that maps cleanly to FFN layers. Attention has dynamic sequence length and flexible memory access patterns, which the GPU handles better.

Step 2 — Export With the Right Compute Unit Setting

Here is the minimal setup to get this working. When exporting with coremltools, use ComputeUnit.ALL and let the compiler make scheduling decisions:

import coremltools as ct model_spec = ct.convert( traced_model, compute_units=ct.ComputeUnit.ALL, # Let the compiler decide minimum_deployment_target=ct.target.iOS17 ) model_spec.save("MyLLM.mlpackage") 

Do not force ANE_ONLY on attention blocks. The docs do not make this obvious, but doing so triggers a silent fallback to CPU and destroys your latency budget. Profile first with Instruments' Core ML template before you override anything.

Step 3 — Choose Your Quantization Strategy

To fit under the 2GB weight threshold, you have two practical options.

4-bit palettization maps each weight to an index in a learned codebook — roughly 4x compression over float16. The ANE loads palettized weights natively on A17 Pro and M-series chips:

import coremltools.optimize.coreml as cto op_config = cto.OpPalettizerConfig(mode="kmeans", nbits=4) config = cto.OptimizationConfig(global_config=op_config) palettized_model = cto.palettize_weights(model_spec, config) 

Block-wise float16 (per-channel or per-group) retains float16 precision per block of N weights. Less aggressive compression, but better for layers sensitive to weight distribution — specifically the first and last projection layers.

The combination that works: 4-bit palettization across the bulk of the model, float16 for the first and last projection layers. A 3B parameter model lands around 1.7GB — under threshold, with headroom left for activations and the KV-cache.

Step 4 — Implement the Streaming Token Loop

Here is the working Swift pattern for token-by-token generation with KV-cache threading:

let config = MLModelConfiguration() config.computeUnits = .all let model = try MLModel(contentsOf: modelURL, configuration: config) var inputTokens: [Int32] = tokenize(prompt) while !stopConditionMet { let inputFeatures = try MLDictionaryFeatureProvider(dictionary: [ "input_ids": MLMultiArray(inputTokens), "attention_mask": currentMask, "past_key_values": kvCache ]) let output = try model.prediction(from: inputFeatures) let nextToken = argmax(output.featureValue(for: "logits")) kvCache = output.featureValue(for: "present_key_values") inputTokens = [nextToken] emit(nextToken) } 

Step 5 — Calculate Your KV-Cache Ceiling Before Shipping

Here is the gotcha that will save you hours. KV-cache memory scales linearly with sequence length:

kv_bytes = 2 × n_layers × n_heads × head_dim × seq_len × dtype_bytes 

For a 32-layer, 32-head model with head_dim=128 in float16 at 2048 tokens:

2 × 32 × 32 × 128 × 2048 × 2 = ~1.07 GB 

Add that to 1.7GB of model weights and you are at ~2.8GB total — well past the jetsam threshold on most iPhone configurations. On iPhone 16 with 6GB LPDDR5, the practical usable context ceiling is roughly 1024–1500 tokens before the OS starts issuing memory pressure notifications.

The fix is static KV-cache allocation: pre-allocate the cache tensor at max sequence length and use a position index to avoid dynamic reallocation, which trips memory fragmentation warnings. Set a hard max_sequence_length in your ML Program spec.

Gotchas

Forcing ANE_ONLY on attention blocks. This is the most common mistake I see. It looks like it should be faster. It is not — the compiler falls back silently to CPU. Always start with ComputeUnit.ALL.

Skipping the KV-cache math. Crashing with jetsam in the background is a one-star review waiting to happen. Run the formula with your model's actual architecture constants before you set a context window length in your UI.

Uniform quantization across all layers. The first and last projection layers are sensitive to weight distribution. Palettizing them at 4-bit degrades output quality noticeably. Keep those in float16.

Testing only on simulator. The ANE does not run in the iOS simulator. All your latency and memory measurements need to come from a physical A17 Pro or later device.

Conclusion

Three constraints, and you have to get all three right: ANE scheduling, quantization strategy, and KV-cache sizing. Use ComputeUnit.ALL and profile before overriding. Default to 4-bit palettization with float16 boundaries. Calculate your cache ceiling with the formula above and enforce it with a hard limit in your ML Program spec.

Further reading:

• Apple CoreML Tools documentation
• Apple Instruments Core ML template guide
• ML Program format specification