Pre-Training

Every fact a model can state, and every fact it cannot, gets decided before any fine-tuning or alignment step ever touches it — pre-training is where that ceiling gets set, once, at a cost almost no team pays itself. It is the first and most expensive rung of the LLM training ladder, and the one stage most developers building on frontier models take entirely on faith. That faith is being tested in 2026: token counts keep climbing past twenty trillion even as the highest-quality raw text needed to keep climbing runs short.

The pre-training reading path: mechanism to trillion-token race

Start with what pre-training is and how LLMs learn language from raw text at scale — it owns the mechanism, next-token prediction repeated across trillions of words, that every other article here assumes you already have. From there, the building blocks of a modern pre-training pipeline turns that mechanism into an engineering process: curation, deduplication, distributed training, checkpoints. Read the technical limits of pre-training in 2026 next to see where that process stops paying off — compute keeps climbing while the highest-quality text supply does not.

When you are ready to run one yourself, the Megatron-LM, DeepSpeed, and NeMo guide gives the current tooling and the parallelism decisions that separate a working run from a wasted week of GPU time. For where the frontier actually sits, the 28-trillion-token race tracks how labs are answering the data-exhaustion problem with scale and synthetic augmentation anyway. Close with the ethical price of training on trillions of tokens — the corpus that makes pre-training work was never fully consented to, and that cost shows up in no benchmark.

How pre-training differs from scaling laws, fine-tuning, and RLHF

• Pre-training is not scaling laws. Pre-training is the run itself — the process that consumes tokens and produces a base model. Scaling laws are the predictive math laid over that process, estimating in advance how much a bigger run should improve before you spend the compute to find out. You do not need the law to pre-train a model, but skipping it means learning its curve the expensive way.
• Pre-training is not an early form of fine-tuning. Fine-tuning starts from a model that already carries pre-training’s knowledge and adjusts behavior on a small curated set; pre-training is what puts that knowledge there in the first place, drawn from a raw, web-scale corpus. Continuing to train an existing checkpoint on more unlabeled text keeps the objective and the data pre-training’s, whatever the marketing calls it.
• Pre-training does not touch preferences. Everything a model prefers to say, and everything it refuses, comes later — from RLHF and the reward model it optimizes against. A behavior problem that traces back to pre-training is a knowledge gap; one that traces to RLHF is a preference problem. The two get diagnosed, and fixed, in completely different places.

Common questions about pre-training

Q: What happens to a pre-training run once it stops — is the model finished? A: No. The run’s last checkpoint becomes the base model everything else starts from. The building blocks of a pre-training pipeline covers how checkpoints get saved, validated, and handed off to whatever fine-tuning or alignment comes next.

Q: Do I need Megatron-LM or DeepSpeed to run a small experimental pre-training job? A: Not necessarily — those tools solve distributed-parallelism problems that only appear at multi-GPU, multi-node scale. The Megatron-LM, DeepSpeed, and NeMo guide explains which of the four pipeline layers actually need that tooling and which do not.

Q: Why do labs keep crossing new token-count records if scaling returns are diminishing? A: Because the race changed targets. GLM-5 and the 28-trillion-token race shows labs now treating raw pre-training scale as a commodity input, while the real competition has moved to data quality and what happens after pre-training ends.

Q: Is training on scraped web text without permission a settled legal question by 2026? A: No. The ethical price of training on trillions of tokens traces ongoing copyright disputes, carbon costs, and consent gaps that pre-training’s scale created and that courts and regulators have not resolved.

Part of the LLM training ladder · closest neighbour: scaling laws. Coming to this from a software background? Start with the story: LLM Training for Developers: Which Instincts Help, Which Mislead.