Pre-Training

What pre-training is

Every large language model starts as a blank slate — billions of numerical weights with no understanding of language, logic, or the world. Pre-training is the process that changes this. It’s the stage where a model consumes enormous volumes of raw text — books, websites, code repositories, scientific papers — and learns to predict what comes next, one token at a time.

Think of it like learning a language by reading every book in a library without a teacher. Nobody tells the model what “grammar” is or explains sentence structure. Instead, the model figures out patterns on its own by repeatedly guessing the next word in a sequence and adjusting its internal weights when it guesses wrong. This approach is called self-supervised learning — the text itself provides the training signal, with no human-labeled examples needed.

Two main approaches emerged early on. According to Radford et al., the 2018 GPT paper demonstrated autoregressive pre-training — the model reads text left to right and predicts the next token at each position. According to Devlin et al., BERT took a different path the following year with masked language modeling: hiding random words and asking the model to fill in the blanks. Today, most frontier LLMs use the autoregressive approach.

According to Kaplan et al., model performance follows predictable power-law relationships with three variables: the number of parameters, the volume of training data, and the amount of compute used. Increasing any of these in the right proportions yields better results — a finding that drives the push toward ever-larger pre-training runs.

Pre-training consumes the vast majority of the total compute budget and can run for weeks or months on thousands of specialized processors. The result is a foundation model — a general-purpose language engine that encodes broad world knowledge but hasn’t learned to follow instructions or behave safely. That comes later, through fine-tuning and alignment techniques like RLHF.

How pre-training is used in practice

When you interact with AI assistants like Claude or ChatGPT, every response draws on knowledge encoded during pre-training. The model’s ability to write code, summarize documents, answer technical questions, or translate between languages all traces back to patterns learned during this foundational phase.

For most people, pre-training matters because it determines what the model knows before any customization. If a model was pre-trained on a corpus rich in programming documentation, it handles coding tasks better out of the box. If the pre-training data skewed toward English text, performance in other languages suffers. The composition of pre-training data directly shapes the capabilities and limitations you experience when using AI products.

For organizations fine-tuning their own models, pre-training sets the ceiling. Fine-tuning can teach new formats, styles, or domain terminology, but it cannot fill gaps in foundational knowledge laid down during pre-training.

When evaluating AI models for your team, ask about the pre-training data composition — not just the parameter count. A model trained on diverse, high-quality data often outperforms a larger model trained on lower-quality sources.

The pre-training reading path: mechanism to trillion-token race

Start with what pre-training is and how LLMs learn language from raw text at scale — it owns the mechanism, next-token prediction repeated across trillions of words, that every other article here assumes you already have. From there, the building blocks of a modern pre-training pipeline turns that mechanism into an engineering process: curation, deduplication, distributed training, checkpoints. Read the technical limits of pre-training in 2026 next to see where that process stops paying off — compute keeps climbing while the highest-quality text supply does not.

When you are ready to run one yourself, the Megatron-LM, DeepSpeed, and NeMo guide gives the current tooling and the parallelism decisions that separate a working run from a wasted week of GPU time. For where the frontier actually sits, the 28-trillion-token race tracks how labs are answering the data-exhaustion problem with scale and synthetic augmentation anyway. Close with the ethical price of training on trillions of tokens — the corpus that makes pre-training work was never fully consented to, and that cost shows up in no benchmark.

How pre-training differs from scaling laws, fine-tuning, and RLHF

• Pre-training is not scaling laws. Pre-training is the run itself — the process that consumes tokens and produces a base model. Scaling laws are the predictive math laid over that process, estimating in advance how much a bigger run should improve before you spend the compute to find out. You do not need the law to pre-train a model, but skipping it means learning its curve the expensive way.
• Pre-training is not an early form of fine-tuning. Fine-tuning starts from a model that already carries pre-training’s knowledge and adjusts behavior on a small curated set; pre-training is what puts that knowledge there in the first place, drawn from a raw, web-scale corpus. Continuing to train an existing checkpoint on more unlabeled text keeps the objective and the data pre-training’s, whatever the marketing calls it.
• Pre-training does not touch preferences. Everything a model prefers to say, and everything it refuses, comes later — from RLHF and the reward model it optimizes against. A behavior problem that traces back to pre-training is a knowledge gap; one that traces to RLHF is a preference problem. The two get diagnosed, and fixed, in completely different places.

When pre-training fits — and when it doesn't

Common questions about pre-training

Q: What happens to a pre-training run once it stops — is the model finished? A: No. The run’s last checkpoint becomes the base model everything else starts from. The building blocks of a pre-training pipeline covers how checkpoints get saved, validated, and handed off to whatever fine-tuning or alignment comes next.

Q: Do I need Megatron-LM or DeepSpeed to run a small experimental pre-training job? A: Not necessarily — those tools solve distributed-parallelism problems that only appear at multi-GPU, multi-node scale. The Megatron-LM, DeepSpeed, and NeMo guide explains which of the four pipeline layers actually need that tooling and which do not.

Q: Why do labs keep crossing new token-count records if scaling returns are diminishing? A: Because the race changed targets. GLM-5 and the 28-trillion-token race shows labs now treating raw pre-training scale as a commodity input, while the real competition has moved to data quality and what happens after pre-training ends.

Q: Is training on scraped web text without permission a settled legal question by 2026? A: No. The ethical price of training on trillions of tokens traces ongoing copyright disputes, carbon costs, and consent gaps that pre-training’s scale created and that courts and regulators have not resolved.

Part of the LLM training ladder · closest neighbour: scaling laws. Coming to this from a software background? Start with the story: LLM Training for Developers: Which Instincts Help, Which Mislead.

Sources

• Radford et al.: Improving Language Understanding by Generative Pre-Training - The original GPT paper establishing autoregressive pre-training for language models
• Kaplan et al.: Scaling Laws for Neural Language Models - Defines power-law relationships between model size, data, compute, and performance