What Does It Take to Run a 30B Model Without a GPU?

Most people interact with large language models through frameworks that hide much of the inference stack. Those frameworks are powerful, but they can make it difficult to see what happens between loading a model and generating the next token.

So I built Quantr, an experimental LLM inference engine written primarily in C.

The project explores quantized model weights, GGUF loading, CPU execution, low-level kernels, context management, and a runtime that can be inspected without making PyTorch or BLAS mandatory dependencies. Its CPU path does not require a GPU.

This is not a claim that a small binary makes every large model fit into a small amount of RAM. The interesting engineering challenge is understanding which parts of inference can be made lighter, which trade-offs are involved, and how to verify that the implementation is doing what it claims.

Repository: github.com/bharqav/quantr-in-c

What Is Quantr?

Quantr is a C-based inference runtime built to explore quantized transformer models in the GGUF format.

The repository brings together several pieces of an inference stack:

• GGUF metadata and tensor loading.
• Memory-mapped model-file access where supported.
• Quantized tensor representations, including Q4_K, Q5_K, Q6_K, and Q8_0 paths.
• Scalar CPU computation and available SIMD kernels.
• Tokenization, model execution, sampling, and context-management code.
• Experimental speculative-decoding and backend-verification utilities.
• Tests for selected numerical operations and runtime components.

The core idea is simple: make the inference path explicit. Instead of treating the runtime as a black box, I wanted to understand the model format, memory layout, numerical operations, and CPU execution decisions directly.

Quantr runs as a normal user-space program under an operating system. I sometimes describe the project as low-level or bare-metal-inspired, but it is not an operating-system-free bare-metal runtime.

Why Build an Inference Engine from Scratch?

An inference engine is not just a matrix multiplication loop. It has to coordinate several layers of work:

• Model representation: parse metadata and understand tensor layouts.
• Memory management: load weights and manage intermediate activations and attention state.
• Numerical computation: implement operations such as normalization, attention, and feed-forward layers.
• Quantization: reduce weight storage while accounting for numerical error and dequantization requirements.
• Hardware execution: use SIMD instructions where available while retaining a fallback path.
• Runtime behavior: handle tokenization, sampling, context limits, and generation.
• Verification: test individual operations and establish whether complete model outputs are correct.

Working in C makes these boundaries visible. It also removes some of the safety nets provided by higher-level frameworks. Tensor shapes, memory ownership, numerical precision, and architecture-specific behavior become your responsibility.

That is what makes the project useful as a systems-learning exercise: every shortcut has a cost, and every performance claim needs evidence.

The Weight and Memory Problem

Model weights are often the largest component of inference memory, but they are not the only component.

For a rough estimate, a model with 30 billion parameters stored at FP16 needs around 60 GB for its weights alone. At an idealized four bits per parameter, the raw weight payload would be around 15 GB before block metadata, tensor-specific formats, and runtime overhead.

That distinction matters because several different optimizations are often discussed as if they all reduce the same quantity.

The important lesson is that file size, virtual address space, resident memory, and total system memory are different measurements. A model file being memory-mapped does not mean the model has no physical-memory cost.

Any claim that a large model runs within a particular RAM budget should be backed by a reproducible test that identifies the exact model file, quantization, context length, hardware, operating system, and memory metric.

Model tiers and memory figures from the original draft

The original article included the following system-tier table. These are reported figures from the draft, not independently verified benchmark results. Keep them labelled that way unless you can reproduce them with the exact model files and test setup.

The original draft also described a sequence of memory reductions. The values below are retained for transparency, but the steps combine different concepts and should not be read as a validated, additive reduction pipeline.

How It Works

1. GGUF Loading and Memory Mapping

GGUF is a model-file format used by tools such as llama.cpp. A loader needs to interpret its metadata and tensor descriptions before the runtime can use the weights.

Quantr contains GGUF parsing and model-loading code, including memory-mapped file access on supported platforms. Memory mapping lets a process address file contents through virtual memory instead of first copying the entire file into a separate allocation.

Conceptually, the flow looks like this:

GGUF model file | v Parse metadata and tensor descriptions | v Map or access tensor data | v Build model/runtime structures | v Run inference operations 

The operating system controls when mapped pages become resident in physical memory. This can reduce eager copying and make loading more flexible, but it does not eliminate the need for RAM. Access patterns, page faults, storage speed, and memory bandwidth still matter.

2. Quantized Weights and CPU Kernels

Quantization stores supported weights with fewer bits than FP32 or FP16 representations. This can reduce the amount of data that must be read from memory, which is especially important for CPU inference.

Quantr includes quantization-related code and CPU kernels for selected operations. It also contains SIMD-specific paths alongside a scalar fallback. The exact formats and operations supported depend on the relevant implementation.

A simplified quantized dot product illustrates the basic idea:

/* * Conceptual outline only. * The actual implementation depends on the quantization format. */ for (int block = 0; block < num_blocks; ++block) { /* Read packed quantized values and their scale metadata. */ /* Compute the block contribution. */ /* Accumulate the result. */ } 

A quantized kernel does not necessarily mean that every operation stays in integer arithmetic from end to end. Some formats require scale handling, conversion, or floating-point accumulation. The implementation must be inspected operation by operation rather than described with a blanket claim that dequantization never occurs.

SIMD can improve throughput by processing multiple values per instruction. It does not guarantee that every backend is faster in every workload. Results depend on the CPU, compiler, memory layout, sequence length, and the operation being measured.

3. Mixture-of-Experts Models

A Mixture-of-Experts (MoE) model uses a routing mechanism to select a subset of experts for a token. This can reduce the amount of expert computation performed per token compared with evaluating every expert.

The key distinction is between active computation and stored parameters. Sparse routing can reduce the work performed during a forward pass, but the inactive experts still belong to the model and their weights still need to be represented in the model file or available storage.

Quantr includes MoE-related routing code. That alone does not establish support for every MoE architecture, nor does it prove that a model with a given total parameter count fits into a particular RAM budget. Those claims require an end-to-end test with the exact checkpoint and configuration.

4. Context Management and the KV Cache

During autoregressive generation, a transformer repeatedly uses information from previous tokens. Many transformer implementations retain attention keys and values in a key-value cache, commonly called the KV cache, so they do not need to recompute all prior attention state for every new token.

The KV cache can become a substantial memory cost as the context grows. Its size depends on factors such as layer count, number of key-value heads, head dimension, token count, batch size, and cache data type.

The original article gave these approximate per-token cache costs:

These figures depend on the model architecture and what the calculation includes. They should not be presented as universal per-token costs. More importantly, support for Q8_0 weight quantization does not prove Q8_0 KV-cache support. The cache's allocation type and update path must be checked separately.

Quantr contains context-window and KV-cache-related code. However, removing old tokens from a token-history structure is not by itself sufficient to guarantee that cached attention state remains consistent. A correct sliding-window implementation must keep token positions, cache entries, and the model's positional-encoding behavior aligned.

Likewise, supporting quantized weights does not imply that the KV cache is quantized. The cache's allocation type and update path must be checked separately. Cache-memory savings should be reported only for a cache format that is actually implemented and exercised by a reproducible test.

5. Experimental Speculative Decoding

Speculative decoding uses a smaller draft model to propose tokens and a target model to verify them. The goal is to reduce the number of expensive target-model steps required to generate a sequence.

Correct speculative sampling is more involved than comparing the draft token with the target model's most likely token. To preserve the target model's sampling distribution, the algorithm needs a mathematically correct acceptance rule and a corresponding rejection-sampling procedure.

Quantr contains experimental speculative-decoding and verification code. That is a useful foundation to investigate, but the presence of a verifier or an acceptance check is not, on its own, proof of standard exact speculative sampling. A stronger claim requires checking the full generation path and validating output distributions or deterministic behavior against a trusted reference under clearly specified decoding settings.

For that reason, speculative decoding should be treated as an experimental component until end-to-end parity has been demonstrated.

How I Test It

Low-level numerical code needs tests, but a passing unit test suite and a fully validated inference engine are not the same thing.

Quantr includes tests for selected components such as numerical operations, quantization and dequantization paths, tokenizer behavior, context accounting, and other runtime utilities.

These tests are useful for catching regressions in the pieces they exercise. They do not automatically establish compatibility with every GGUF model, every architecture, or every supported CPU backend.

A robust verification plan needs multiple levels:

Generated text matching is useful evidence, but it is not equivalent to proving that all logits are bit-identical. Sampling settings, seeds, tokenization, numerical precision, and implementation details all affect comparisons.

For meaningful external comparisons, the model file, prompt, context length, decoding parameters, reference version, and output comparison method should be documented.

Test cases listed in the original draft

The source article listed 15 passing tests. The table below preserves the names and status reported there. Treat these as reported test results until the current test suite is run and the output is captured from the repository version being published.

Reported summary: 15 of 15 tests passed. This summary records what the original draft says; it is not a substitute for rerunning the tests.

Backend and reference comparison claims

The original draft also reported the following comparison:

The strongest version of this claim would include the exact command, model checksum, prompt, decoding configuration, raw comparison output, and the script that ran each backend.

Benchmarking Responsibly

Performance numbers are only useful when the test conditions are clear. A benchmark should identify:

• The exact model and GGUF file.
• The quantization format.
• CPU model, core count, and supported SIMD instructions.
• Operating system and compiler flags.
• Prompt length and generated-token count.
• Context length and KV-cache data type.
• Whether the reported memory is peak RSS, working set, virtual memory, or another metric.
• Whether the measurement includes model loading, prompt processing, or generation only.

I am deliberately not presenting unverified model-size, RAM, or tokens-per-second figures as established results here. In particular, a claim that a 30B-parameter model runs in a few gigabytes of RAM needs a reproducible end-to-end benchmark for that exact model. Sparse MoE execution and memory mapping do not, by themselves, establish such a result.

A useful next step for Quantr is to add a benchmark harness that records configuration and environment details automatically, then compare its results against a trusted implementation.

Performance figures listed in the original draft

These values are included because they appeared in the original article. They should be labelled as reported measurements pending reproducibility, rather than treated as independently confirmed results.

The original article also included this thread-scaling result for TinyLlama:

The original interpretation was that scaling flattens beyond eight threads, potentially because inference becomes memory-bandwidth-bound. That is a plausible explanation, but confirming it requires a controlled benchmark and ideally memory-bandwidth profiling. The table alone does not establish the cause.

For publication, attach the test environment to every benchmark table: CPU model, RAM, operating system, compiler and flags, model filename and checksum, prompt length, generated-token count, context length, cache type, warm-up policy, and whether loading and prompt processing are included. Without those details, these figures are useful as historical project notes but difficult for others to reproduce.

Building and Exploring the Code

Start by cloning the repository:

git clone https://github.com/bharqav/quantr-in-c.git cd quantr-in-c 

Build and test commands can vary by platform and toolchain, so use the targets defined in the repository's current Makefile rather than assuming a particular executable name. On Windows, the available make tool and generated executable suffix may differ from Linux or macOS.

The main areas of the codebase are organized around responsibilities such as:

quantr-in-c/ ├── include/ # Public headers and data structures ├── src/ # Model loading, kernels, runtime, and supporting code ├── tests/ # Tests for selected components ├── scripts/ # Verification and development utilities └── Makefile # Build targets 

The exact files and targets may evolve as the project changes. Check the repository before relying on a command or a specific implementation detail.

When reading the code, I recommend following the path from model loading to a single inference step: inspect how metadata is parsed, how tensors are represented, how an operation is dispatched to a kernel, and how its result reaches the next layer. That path makes the system easier to reason about than starting with the terminal interface alone.

What I Learned

Correctness needs to be designed in

A low-level engine can appear to work while still mishandling edge cases in tensor layouts, quantization, cache updates, or sampling. Unit tests help, but the goal is to connect them to end-to-end tests with explicit invariants.

Memory optimization has several meanings

Reducing a file's size, reducing allocations, avoiding a copy, and reducing resident memory are related but different goals. The distinction becomes especially important when discussing large models and memory-mapped files.

CPU inference is a systems problem

Quantization, memory access patterns, cache behavior, vectorization, threading, and compiler choices all interact. Counting arithmetic operations alone is not enough to explain real-world performance.

A feature is not the same as a validated guarantee

Code for a quantized format, a SIMD path, or speculative decoding is the start of the engineering work, not the end. Each feature needs tests that exercise its real execution path and clearly define what success means.

What Comes Next?

The next stage of Quantr is less about adding bold claims and more about closing the gap between implementation and evidence.

My priorities are:

1. End-to-end model validation: document which model architectures and GGUF variants are actually supported.
2. Reproducible benchmarks: record model files, hardware, compiler settings, prompt sizes, latency, and memory measurements.
3. Backend verification: compare the paths that are genuinely built and exercised, using clear numerical tolerances.
4. KV-cache correctness: validate cache allocation, updates, context eviction, and positional consistency together.
5. Speculative-decoding validation: verify the full acceptance and rejection procedure against a trusted reference.
6. Build documentation: keep commands, platform support, and test instructions synchronized with the current repository.

These improvements would make it easier for other developers to reproduce the results, understand the runtime's boundaries, and contribute without having to infer which claims are experimental.

The Takeaway

Quantr is my attempt to understand LLM inference closer to the metal: how model files are represented, how quantized weights are processed, how CPU kernels are dispatched, and how a runtime coordinates all of these pieces.

The project is experimental, and its strongest performance and correctness claims need reproducible evidence. That is part of the engineering process, not a reason to hide the work. A useful inference engine is not defined by how impressive its headline sounds. It is defined by the behavior that can be measured, explained, and reproduced.

The code is available here: github.com/bharqav/quantr-in-c

If you explore it, I would be especially interested in feedback on model compatibility, kernel verification, memory accounting, and benchmark methodology.