TL;DR: You almost never need to pre-train a model. The economical path for small teams and researchers is to take a strong open base model → fine-tune it cheaply (LoRA/QLoRA) on your high-quality data → or skip training entirely and use RAG. Pre-training from scratch is for the few labs with nine-figure budgets.
This is a conceptual + strategic guide. It covers:
- How machine-learning models (especially large language models) are actually trained.
- How to make training economical.
- How the Chinese labs (DeepSeek, Kimi/Moonshot, Qwen/Alibaba) train differently from the Western frontier labs (OpenAI, Anthropic/Claude, Google/Gemini).
- How all of those differ from agentic tools like Kiro.
- Where specialized research projects (like NeuroJen) sit on that spectrum.
- How the Prima Mente Alzheimer's Translation Challenge relates to this ecosystem.
Facts about third-party models were verified against public sources (technical reports, model cards, vendor docs) as of October 2026. The field moves fast; treat version-specific numbers as "accurate at time of writing."
Table of contents
- How ML models are trained (the fundamentals)
- The LLM training pipeline, stage by stage
- What training actually costs — and how to make it economical
- The Chinese labs: DeepSeek, Kimi, Qwen
- The Western frontier labs: OpenAI, Anthropic, Google
- How they differ — a side-by-side
- Where Kiro fits (it isn't a model maker)
- Where specialized projects fit
- The Prima Mente Alzheimer's Translation Challenge
- How the challenge relates to this project
- Glossary
1. How ML models are trained
At its core, "training" a model means adjusting millions-to-trillions of numbers (parameters/weights) so the model's predictions get closer to the right answers.
The loop is always the same:
1. Forward pass — Feed input in, model produces an output (a prediction). 2. Loss — Compare the prediction to the desired answer; compute an error number ("loss"). Lower = better. 3. Backward pass — "Backpropagation": compute how each weight contributed to the error (the gradient). 4. Update — Nudge each weight slightly in the direction that reduces the loss (an "optimizer" like AdamW does this). 5. Repeat — Millions of times over billions of examples. A few ideas that matter for everything below:
- Parameters (weights): The model's learned knowledge. "A 7B model" = 7 billion parameters. More parameters ≈ more capacity, but also more compute and memory.
- Tokens: Text is chopped into tokens (~¾ of a word each). Models are trained on trillions of tokens. "14.8T tokens" means 14.8 trillion.
- Compute: Measured in GPU-hours or FLOPs. This is the dominant cost.
- The scaling laws: Empirically, performance improves predictably as you scale model size, data, and compute together. This is why frontier models got huge. But bigger is not automatically better per dollar — which is the whole story of Sections 3–4.
2. The LLM training pipeline
Modern large language models are built in phases, each with a different purpose:
Phase A — Pre-training (the expensive one)
The model reads a vast corpus (web text, books, code — trillions of tokens) and learns to predict the next token. This is self-supervised: no human labels needed, the "answer" is just the next word in the text. This phase creates a base model that has broad knowledge but isn't yet helpful or safe. It consumes ~95%+ of the total training compute.
Phase B — Supervised Fine-Tuning (SFT) / instruction tuning
The base model is trained on curated input→output pairs (e.g. "Question → good answer", "Instruction → correct completion"). This teaches it to follow instructions and respond in a useful format. Much cheaper than pre-training — thousands to millions of examples, not trillions of tokens.
Phase C — Alignment / preference optimization
Make the model helpful, honest, and harmless. Techniques:
- RLHF (Reinforcement Learning from Human Feedback): Humans rank answers; a reward model learns their preferences; the LLM is optimized against it (classically with PPO).
- DPO (Direct Preference Optimization): A simpler, cheaper alternative to RLHF that skips the separate reward model.
- Constitutional AI (Anthropic): The model critiques/revises its own answers against a written set of principles ("a constitution"), reducing the human-labeling burden.
Phase D — Reasoning training (the 2024–2026 shift)
Teach the model to "think" before answering — produce a long chain-of-thought, then the final answer. This is often done with reinforcement learning on verifiable rewards (e.g. math/code problems where correctness is checkable). DeepSeek-R1 and OpenAI's "o" series popularized this. It dramatically improves hard reasoning without making the base model bigger.
3. What training costs
Why it's expensive
Frontier pre-training needs thousands of top-tier GPUs running for months. A single frontier run can cost tens to hundreds of millions of dollars in compute alone — plus data, salaries, and failed runs. This is why only a handful of organizations pre-train frontier models from scratch.
The levers that make training economical
| Lever | What it does | Who uses it |
|---|---|---|
| Mixture-of-Experts (MoE) | Only activate a subset of the network per token (e.g. 37B of 671B). Huge capacity, fraction of the compute per token. | DeepSeek, Kimi, Qwen, Mixtral |
| Low precision (FP8/INT4) | Train/run in 8-bit (or 4-bit) instead of 16/32-bit → less memory, faster math. | DeepSeek (FP8 pre-training), most quantized inference |
| Parameter-Efficient Fine-Tuning (LoRA/QLoRA) | Freeze the base model; train tiny "adapter" matrices (~0.1–1% of weights). Fine-tune a big model on one GPU. | Most open-source fine-tuning, specialized research |
| Quantization | Store weights in 4-bit for inference/training → fits bigger models on smaller GPUs. | QLoRA, llama.cpp, Unsloth |
| Distillation | Train a small "student" model to imitate a big "teacher". Cheap models that punch above their size. | DeepSeek-R1-Distill, many small models |
| Better optimizers / stability tricks | Converge faster / avoid wasteful failed runs (e.g. Muon, MuonClip). | Kimi K2 (MuonClip) |
| Data quality over quantity | Clean, de-duplicated, well-curated data trains better per token. | Everyone serious about results |
| Retrieval (RAG) instead of training | Put knowledge in a searchable store and retrieve it at query time rather than baking it into weights. Near-zero training cost. | Most production deployments |
| Free/cheap compute tiers | Kaggle (free T4 GPUs 30h/wk), Colab, Modal credits, spot instances. | Students, small teams, researchers |
The single most important beginner lesson
You almost never need to pre-train. The economical path is: take a strong open base model → fine-tune it cheaply (LoRA/QLoRA) on your high-quality data → or skip training entirely and use RAG. Pre-training from scratch is for the few labs with nine-figure budgets. Everyone else builds on top.
4. The Chinese labs
The defining theme of the Chinese frontier labs is efficiency and openness — achieving near-frontier quality at a fraction of the reported cost, and releasing open weights (you can download and run the model yourself).
DeepSeek (DeepSeek-V3 / R1)
- Architecture: Mixture-of-Experts — 671B total parameters, ~37B active per token. You get the capacity of a huge model at the compute of a ~37B one.
- Scale & cost: Pre-trained on 14.8 trillion tokens using FP8 (8-bit) training, in ~2.788 million H800 GPU-hours. The widely-cited figure is a ~$5.6M compute cost for the main run — one to two orders of magnitude below typical Western frontier estimates. (Caveat: this is the final run's compute, not total R&D, and the comparison is imperfect — but the efficiency is real.)
- R1 (reasoning): Trained reasoning ability largely through reinforcement learning (GRPO — Group Relative Policy Optimization). Famously, R1-Zero was trained with RL and no supervised cold-start, showing reasoning can emerge from RL alone; the shipped R1 adds a small SFT cold-start for readability.
- Openness: Open weights under a permissive (MIT) license; detailed technical reports. This is the opposite of the closed Western approach.
Kimi (Moonshot AI — Kimi K2)
- Architecture: MoE — 1 trillion total parameters, ~32B active; 384 experts, 8 active per token.
- Training innovation: The MuonClip optimizer — the Muon optimizer plus a "QK-clip" technique to keep training stable at trillion-parameter scale (training instabilities are a major hidden cost; fixing them saves wasted runs).
- Focus: Agentic tool-use — built to call tools and act, not just chat. Notably, base K2 is not a chain-of-thought reasoning model by default; it's optimized for fast agentic action.
- Openness: Open weights.
Qwen (Alibaba — Qwen3 family)
- A broad open-weight family: dense models (every parameter active) and MoE variants, across many sizes (small enough for a laptop up to very large).
- Strong on multilingual, coding, and agentic tasks; widely used as a base for downstream fine-tuning because of the permissive licensing and size range.
The common thread
MoE for efficiency + low-precision training + open weights + published methods. The strategic bet: openness drives adoption, and efficiency makes frontier-ish capability affordable.
5. The Western frontier labs
OpenAI (GPT series)
Proprietary, closed weights, API-only. Pioneered the modern RLHF-aligned assistant (InstructGPT → ChatGPT) and the reasoning-model wave (the "o" series). Very large (undisclosed) training budgets; strong product/ecosystem integration.
Anthropic (Claude)
Proprietary, closed weights, API-only. Known for Constitutional AI (aligning the model against a written set of principles with less human labeling) and a strong safety/reliability emphasis — which is why Claude is a popular default for agentic coding (it holds intent across long, multi-step tasks).
Google (Gemini)
Proprietary, closed weights, API-only, deeply multimodal (text, images, audio, video) and integrated across Google's products. Backed by Google's own TPU hardware and enormous data/compute.
Important nuance: Google also publishes the open-weight Gemma family (including the medical MedGemma that many research projects build on). So "Google" spans both closed frontier (Gemini) and open small models (Gemma).
The common thread
Closed weights, API access, large undisclosed budgets, heavy investment in alignment and safety, and productization. You rent capability through an API rather than owning the model.
6. How they differ
| Dimension | Chinese labs (DeepSeek/Kimi/Qwen) | Western frontier (OpenAI/Anthropic/Google) |
|---|---|---|
| Weights | Open — download & self-host | Closed — API only (except Google's Gemma) |
| Headline strategy | Efficiency + openness | Capability + product + safety, kept proprietary |
| Architecture trend | Aggressive MoE (sparse) | Mix; often undisclosed |
| Training precision | FP8 pre-training (DeepSeek) | Undisclosed; generally higher precision historically |
| Reported cost | Dramatically lower (e.g. DeepSeek ~$5.6M run) | Much higher (tens–hundreds of $M), undisclosed |
| Reasoning | RL-centric (GRPO; R1-Zero showed RL-only works) | RLHF + dedicated reasoning models (o-series) |
| Alignment | Lighter / varies | Heavy (RLHF, Constitutional AI, red-teaming) |
| You can fine-tune it? | Yes (open weights) | No (closed) — you can only prompt/call it |
| Transparency | Detailed technical reports | Limited; cards + marketing |
The practical upshot for a small team: Open-weight Chinese (and Google Gemma) models are the ones you can actually download, fine-tune, and own. Closed Western models are things you call.
7. Where Kiro fits
Kiro is not a model maker — it's an agentic development tool (an IDE and CLI). This is a common point of confusion, so it's worth stating plainly:
- Kiro does not train its own foundation models. It runs on Amazon Bedrock and gives you access to other people's models: Anthropic's Claude (its historical default), OpenAI's GPT tiers, and open-weight models like DeepSeek and Qwen.
- Its "Auto" mode is a router: it picks the best available model for a given task based on complexity, latency, and cost. That orchestration — plus spec-driven workflows, tools, and a graph-based state engine — is Kiro's value-add, not the model weights.
So on our map: the labs in Sections 4–5 make the models; Kiro orchestrates them.
Model makers (train) Orchestrators (route+act) Domain fine-tuners (specialize) OpenAI, Anthropic, Kiro (agentic IDE/CLI on Specialized research (e.g. Google, DeepSeek, → Bedrock; routes to Claude/ → MedGemma + LoRA on Moonshot, Alibaba GPT/DeepSeek/Qwen via "Auto") curated scientific data) 8. Where specialized projects fit
Specialized research projects (like NeuroJen) sit deliberately at the economical, specialized end of everything above.
- Base model: MedGemma-4B (a medical-domain model in Google's open Gemma family) — small, open, and already medically literate. Not pre-trained by us.
- Method: LoRA supervised fine-tuning via Unsloth, on a single free Kaggle T4 GPU (4-bit quantized). This is Section 3's economical toolkit applied end to end: open base + PEFT + quantization + free compute.
- The novelty — "answer-withholding" tasks: The training signal hides the field we want the model to predict, so it must reason about the body-to-brain link rather than reformat text.
- Stage 1 before any training — GraphRAG: Retrieval over a Neo4j knowledge graph, which gives value with zero GPU cost and acts as the baseline the fine-tuned model must beat.
- Honest evaluation: The fine-tuned model is actually queried and judged by an LLM-judge rubric.
In one sentence: Where DeepSeek/OpenAI/etc. spend millions to pre-train general models, specialized projects spend ~$0 to specialize an existing open model for one disease — exactly the right strategy for a small research team. We are a consumer and specializer of the ecosystem, not a frontier trainer.
9. The Prima Mente Challenge
Source: https://www.primamente.com/challenges/ (verified Oct 2026).
Prima Mente is an AI-biology company (London/San Francisco) building epigenomic foundation models — large models trained on how genes are switched on/off across cell types (their model line is called Pleiades). They won the $1M ADDI Alzheimer's Insights AI Prize, connecting them directly to the AD Data Initiative ecosystem.
The Alzheimer's Translation Challenge
A global, free AI competition to find new therapeutic hypotheses for Alzheimer's, run with the Alzheimer's Disease Data Initiative (ADDI) and partners (NVIDIA, Hugging Face, Nebius, Prime Intellect, Goodfire, Boltz, and more).
The data: A brand-new open-source cellular atlas of ~150 million cells — neurons, astrocytes, and microglia. Released in batches, accessed via AD Workbench and Prima Mente's modelling platform.
Two stages:
- Stage 1 (build & evaluate): Every participant builds AI models on the atlas, assessed on a curated set of evals — general benchmarks, harder tasks such as phenocopy (predicting whether a perturbation reproduces a disease-like cell state), and community-proposed evals. Strongest results are shortlisted.
- Stage 2 (hypothesize & validate): Shortlisted participants each propose one new therapeutic hypothesis with its anticipated molecular/functional phenotypes. Finalists' hypotheses are tested in Prima Mente's wet lab — closing the loop "from biology to target to function."
Timeline (as published):
| When | Milestone |
|---|---|
| October 2026 | Registration opens (atlas generation underway) |
| November 2026 | Community launches (Q&A, teammates) |
| December 2026 | Prizes announced |
| Spring 2027 | Competition starts (dataset released; model-building + evals) |
| Summer 2027 | Hypothesis validation (finalists' hypotheses tested in the wet lab) |
Key terms: Free to enter; open to anyone (no biology background required — "set up so ML researchers can take part"); Prima Mente takes no rights to your model or discoveries; you may suggest a gene/compound to test at registration.
10. How the challenge relates to this project
This is the strategically important part. Specialized projects and the Prima Mente challenge are complementary layers of the same translational-research stack, and the challenge is a natural proving ground for this thesis.
The connection in one picture
POPULATION / LITERATURE LAYER CELLULAR / MECHANISM LAYER Specialized Project Prima Mente challenge Mines papers → extracts structured 150M-cell atlas (neurons, astrocytes, "peripheral factor → mechanism → microglia) → model perturbations → central pathology" hypotheses phenocopy / therapeutic hypotheses \ / \ / ▼ ▼ A testable hypothesis from the literature becomes a perturbation to evaluate in cells, and (if shortlisted) in Prima Mente's wet lab. Concrete ways they connect
Hypothesis generation vs. validation: The specialized project generates hypotheses about the body-to-brain axis. The challenge is literally a hypothesis-validation engine (Stage 2 → wet lab). A finding like "chronic gut inflammation → gut-brain signalling → microglial activation in the hippocampus" is exactly the kind of specific, testable claim the challenge asks for — and the atlas contains hippocampal microglia to test it against.
Shared infrastructure (AD Workbench): The challenge data is served through the AD Workbench — the same ADDI cloud environment researchers are applying to use. Work done for one directly builds capability for the other.
Shared ecosystem partner (Hugging Face): Hugging Face is a challenge partner and the standard hub for models/datasets. The tooling overlaps significantly.
"Translation" is the shared mission: The challenge is named the Translation Challenge because it moves insights from computation to lab-validated biology — which is precisely the "body-to-brain" aim, one layer up (literature) from Prima Mente's layer (single-cell).
The register-a-perturbation hook: At registration you can submit a gene or compound you'd like tested. The structured hypotheses produced by the literature-mining front end are a ready-made source of candidate perturbations to propose — turning output directly into challenge input.
Why this matters for the roadmap
Participating (or at least aligning with the challenge) gives:
- A concrete downstream validation path for hypotheses (otherwise they're just text).
- Access to cellular-level data that complements literature-level data.
- Free compute and a community (NVIDIA/Nebius/Prime Intellect credits via the challenge).
- A credible, deadline-driven milestone (Spring 2027) to aim the pipeline at.
Bottom line: The literature-mining front end and the cellular-and-wet-lab back end are two halves of the same "from papers to validated targets" pipeline.
11. Glossary
| Term | Plain meaning |
|---|---|
| Parameter / weight | One of the many numbers a model learns. More = more capacity + cost. |
| Token | A chunk of text (~¾ word) the model reads/produces. |
| Pre-training | The big, expensive phase: learn language by predicting the next token on trillions of tokens. |
| SFT | Supervised Fine-Tuning: teach a base model to follow instructions with input→output pairs. |
| RLHF | Reinforcement Learning from Human Feedback: align a model to human preferences. |
| DPO | Direct Preference Optimization: a simpler, cheaper RLHF alternative. |
| Constitutional AI | Anthropic's method: the model self-critiques against written principles. |
| MoE | Mixture-of-Experts: activate only a subset of the network per token → big capacity, low per-token compute. |
| LoRA / QLoRA | Train tiny adapter weights (optionally on a 4-bit model) instead of the whole model → cheap fine-tuning. |
| Quantization | Store/run weights in fewer bits (FP8/INT4) → less memory, faster, cheaper. |
| Distillation | Train a small "student" to imitate a big "teacher". |
| GRPO | Group Relative Policy Optimization — the RL method behind DeepSeek-R1's reasoning. |
| MuonClip | Kimi K2's optimizer (Muon + QK-clip) for stable trillion-parameter training. |
| RAG / GraphRAG | Retrieval-Augmented Generation: fetch knowledge at query time instead of baking it into weights (GraphRAG uses a knowledge graph). |
| Open vs closed weights | Open = you can download & fine-tune the model; closed = API access only. |
| Base vs instruct model | Base = raw pre-trained; instruct = after SFT/alignment, ready to follow instructions. |
| Phenocopy | When a perturbation makes a cell look/behave like it has a disease — a key challenge eval. |
