Hey developers!
My name is Ayushman Singh Rathore, and over the past several months I have been building VeloGrad (v1.0.0), a native-first machine learning and autoregressive inference runtime written from scratch in modern C++20, Rust, and CUDA.
GitHub Repository: https://github.com/IDKLETSCODESOMETHIN/VeloGrad
Why I Built It: PyTorch is great for research, but running modern long-context LLMs in production usually requires duct-taping multiple heavy frameworks together (vLLM, DeepSpeed, TensorRT-LLM). This carries Python GIL stalls, massive multi-gigabyte dependencies, and severe memory fragmentation.
I wanted to explore how far we could push runtime efficiency by designing an engine from bare-metal primitives:
Static Arena Allocator: Instead of dynamic heap allocations during inference, memory is managed via a contiguous arena aligned to 64-byte hardware cache lines (alignas(64)), eliminating GC stalls and latency jitter.
Virtual Paged Attention: We break the KV-cache into discrete physical pages (similar to OS virtual memory paging), reducing KV-cache VRAM usage by ~60% on long contexts without memory fragmentation.
Universal 1-Command Converter (velograd-convert): A zero-dependency binary parser that translates between Hugging Face Safetensors, PyTorch checkpoints, and GGUF v3 (for Ollama / llama.cpp) with no external packages.
PyTorch Drop-in API: Familiar .forward(), .backward(), and autograd syntax with array exchange protocols for NumPy, JAX, and DLPack.
Try it on local hardware in 30 seconds: You can clone the repository, install it locally, and run our universal test suite in under a second:
git clone https://github.com/IDKLETSCODESOMETHIN/VeloGrad.git
cd VeloGrad pip install -e . python tests/test_velograd_universal_compat.py
The project is completely open source under AGPLv3: https://github.com/IDKLETSCODESOMETHIN/VeloGrad
