How to Build an LLM A/B Testing Pipeline with Braintrust, Langfuse, and Promptfoo in 2026

You shipped a prompt change on Friday. By Monday, support tickets were up and you had no data explaining why. You reverted. Four hours of investigation later, you still couldn’t say whether the new prompt caused the regression or whether something else shifted. That’s not a deployment problem — that’s a missing experiment spec.

The fix isn’t more monitoring after the fact. It’s building a pipeline that forces a scoring contract before anything changes.

Before You Start

You’ll need:

• A Braintrust account (Starter tier is free — includes 10k scores and unlimited seats per Braintrust’s pricing page)
• A Langfuse instance — cloud or self-hosted, v2 or later only
• Promptfoo installed via npm (Community tier, free — all providers and LLM eval features included)
• Basic familiarity with LLM Observability and Prompt Versioning And Management

This guide teaches you: how to decompose prompt experimentation into three distinct concerns — offline scoring, production traffic routing, and CI regression gating — and specify each one before writing a single line of integration code.

When Prompt Changes Go Unreported

Here’s the pattern that plays out every week. A developer has a better prompt idea. They update the system prompt in the app config, deploy on a Friday afternoon, and move on. No baseline metric was captured. No variant label was assigned. No scoring function was defined.

By Tuesday, something looks off. Response quality seems different — or doesn’t. Nobody can tell because there was no measurement. The change gets reverted “just to be safe.”

That was an experiment. It just had no spec, no measurement, and no outcome. A well-specified A/B Testing for LLMs pipeline prevents exactly this by making the scoring contract a prerequisite for any deployment.

Step 1: Assign Each Tool a Job

Teams fail here first. They pick one tool and push it to cover everything. Braintrust for traffic routing. Langfuse for scoring. Promptfoo for something undefined. None of them are configured correctly because none have a clear scope.

The answer is a three-way split with zero overlap:

• Braintrust — offline Online Experimentation engine. Creates immutable snapshots per eval run so you can compare variant A against variant B across runs (Braintrust Docs). Supports LLM-as-a-judge, code scorers, classifiers, and human review.
• Langfuse — production observability plus prompt management and traffic splitting. Labels route live requests to specific prompt versions. Tracks latency, token usage, cost per request, and custom eval scores per variant.
• Promptfoo — offline regression testing in CI/CD. Catches prompt quality drops before they reach production. Runs against a golden test set on every pull request.

The Architect’s Rule: Each tool owns one stage. Braintrust scores. Langfuse routes. Promptfoo gates. If you’re asking one tool to do two of these jobs, your pipeline has a spec gap.

Write down this assignment before configuring anything. When a teammate asks “which tool shows cost per variant?” the answer should be instant — Langfuse — without opening three dashboards.

Step 2: Define What “Better” Means

This is the step most teams skip. They wire up the tools, split traffic, and discover three days in that they never agreed on a winner condition. By then, the data is real but the interpretation is contested.

Your scoring contract needs three things defined upfront:

Primary metric — the single number that determines which variant wins. Task success rate. Answer relevance score. Refusal rate on edge inputs. Pick one. Braintrust compares this across experiment snapshots. If you cannot name it before the experiment starts, stop here.

Guardrail metrics — the metrics that must not regress even if the primary metric improves. Latency. Cost per request. Token usage. If variant B is more accurate but significantly slower, that is not a win unless your spec explicitly accepts that trade-off. State the acceptable regression bounds in writing.

Minimum sample size — how many requests each variant needs before you call a winner. Langfuse tracks metrics per label continuously; you decide when the data is sufficient. The exact number depends on your traffic volume and how sensitive your primary metric is — but you must name a specific threshold before traffic starts. Teams that leave this unspecified keep running experiments indefinitely because “we might need more data.”

Scoring contract checklist:

• Primary success metric defined and programmatically measurable
• Guardrail metrics listed with explicit regression bounds
• Minimum sample size or time window stated
• Scoring method selected: LLM-as-a-judge, code scorer, or human review
• Named person or role responsible for calling the winner

The Spec Test: If you cannot complete “we declare variant B the winner when {primary metric condition} AND {guardrail conditions} hold over {sample size}” in one sentence, the scoring contract is not done.

Step 3: Wire Prompt Versions and Route Live Traffic

Langfuse handles Traffic Splitting through labels. Two prompt versions live in Langfuse version control. You assign prod-a to the baseline and prod-b to the challenger. Each version is an immutable snapshot — rolling back means reassigning the label, not deleting the version (Langfuse Docs).

One detail that surprises almost every team: traffic splitting is client-side. Langfuse does not run a server-side router that probabilistically assigns requests. Your application fetches both prompt versions from Langfuse and picks one with random.choice() in Python or Math.random() in JavaScript. The selected label gets attached to the trace. That’s it.

Setup checklist:

• Two prompt versions created in Langfuse prompt management
• Labels assigned: prod-a for baseline, prod-b for challenger
• Application code fetches both versions on each request and randomizes selection
• Each trace tagged with the selected variant label before the request is sent
• Langfuse dashboard configured to compare latency, token usage, and cost per label

Langfuse tracks these metrics per variant automatically once the label is on the trace. What it does not provide: weighted routing at the platform level. If you want a gradual rollout — routing a small fraction of requests to variant B initially — you control that probability in your application code, not in the Langfuse UI.

If you have the option to run Shadow Testing before the live split, use it. Send production requests through both prompt versions in parallel, score the shadow responses offline in Braintrust, and use that data to decide whether a live traffic split is warranted at all. The live split is the riskiest moment; shadow testing moves that risk offline.

Step 4: Score Variants and Call the Winner

This is where Braintrust earns its place. Pull a representative sample of production traces from each variant label. Run them through your scoring pipeline. Compare results in an immutable experiment snapshot that cannot change after the fact (Braintrust Docs).

Validation checklist:

• Variant A and Variant B evaluated on the same input set — failure looks like: comparing on different traffic profiles, which produces unmatchable baselines
• Primary metric compared between snapshots — failure looks like: no meaningful difference after minimum sample size, which means the experiment is inconclusive, not a win
• Guardrail metrics checked — failure looks like: latency or cost regression exceeding your stated bounds
• Promptfoo CI gate passed — failure looks like: score below golden test threshold on the current PR, blocking the label reassignment
• Winner condition met in writing — primary metric up, guardrails held, minimum sample reached

Promptfoo closes the pre-production loop. While Braintrust handles live eval, Promptfoo runs your golden test set on every pull request to any file that touches prompt configuration — catching degradations before they ever reach the live split. With 22.7k GitHub stars as of June 2026, it is among the most widely adopted eval CLIs in active LLMOps pipelines (Promptfoo GitHub). Note: Promptfoo was acquired by OpenAI in March 2026 (OpenAI Blog) and remains MIT-licensed and open source. Avoid forward-looking assumptions about roadmap integration with OpenAI infrastructure — the post-acquisition product direction has not been published.

Compatibility & SDK notes:

• Langfuse SDK < v2.0.0 (BREAKING): Cloud connectivity breaks on Langfuse SDK versions below 2.0, in effect since November 2024. Deprecated v2 endpoints will be removed in a future v3 release. Upgrade to Langfuse SDK v2+ before connecting to any cloud instance (Langfuse Docs).
• Braintrust Python SDK git_diff (v0.20.0+): Automatic git diff logging was removed in v0.20.0. Re-enable it in Settings > Logging if you rely on commit tracking for experiment reproducibility (Braintrust Changelog).
• Braintrust bundler plugins: braintrustVitePlugin, braintrustWebpackPlugin, and related exports were renamed. Check the changelog before upgrading if you use bundler integrations.
• Braintrust API keys: The POST /v1/api_key endpoint was removed in May 2026, then re-enabled for service tokens in June 2026. Use the Braintrust UI for personal API key creation (Braintrust Changelog).

Common Pitfalls

Pro Tip

The decomposition pattern in this guide — assign scope, define success, gate before shipping — applies to every AI system component you touch. When you swap a reranker, add a new retrieval step, or change a LLM Cost Management strategy, the same three questions apply: what is this component’s job, what does improvement look like, and what blocks a bad result from reaching production?

The experiment spec you write for prompt A/B testing is the same mental model that keeps your Model Registry from becoming a graveyard of unverified swaps. Build the habit once. Apply it everywhere.

Frequently Asked Questions

Q: How to build an LLM A/B testing pipeline step by step in 2026? A: Assign tools to non-overlapping jobs first: Braintrust for offline scoring, Langfuse for production traffic splitting, Promptfoo for CI regression gating. Before touching any tool, write a scoring contract — primary metric, guardrail metrics, minimum sample size. Then create two prompt versions in Langfuse, split traffic client-side in your application code, run Braintrust evals on sampled traces, and let Promptfoo block any prompt PR that regresses your golden test set. The pipeline only produces valid experiment results when you define the winner condition before traffic starts flowing — not after.

Q: How to configure prompt versioning and traffic splitting for LLM experiments in production? A: In Langfuse, create two prompt versions and assign labels (prod-a for baseline, prod-b for challenger). Your application fetches both versions and uses random.choice() (Python) or Math.random() (JavaScript) to pick one per request — Langfuse has no built-in server-side router. Tag each trace with the selected label. Langfuse then aggregates latency, token usage, cost, and custom eval scores per label. Weighted rollouts (routing a fraction of requests to variant B for a gradual ramp) require adjusting the sampling probability in your application code. Version history in Langfuse is immutable; rolling back means reassigning the label to a prior version, not deleting the challenger.

Your Spec Artifact

By the end of this guide, you should have:

• A tool assignment map — Braintrust for scoring, Langfuse for routing, Promptfoo for CI gating — with zero overlap between responsibilities
• A written scoring contract — primary metric, guardrail metrics with regression bounds, minimum sample size, and winner condition in one sentence
• A validation checklist — what a passing experiment looks like at each stage, with specific failure symptoms for each check

Your Implementation Prompt

Use this prompt in Claude Code, Cursor, or Codex to generate the integration scaffold for your pipeline. Replace every bracketed placeholder with values from your scoring contract before running it.

You are a senior ML engineer helping me build an LLM A/B testing pipeline.

SYSTEM UNDER TEST:
- Application: [what the LLM does — e.g., "customer support chat that classifies severity and drafts a reply"]
- Language and LLM provider: [e.g., Python 3.11, OpenAI gpt-4o]
- Prompt storage: Langfuse ([cloud or self-hosted], SDK v2+)
- Scoring platform: Braintrust ([Starter / Pro / Enterprise])
- CI eval tool: Promptfoo (Community tier, MIT license)

VARIANT SPECIFICATION:
- Variant A (baseline): [Langfuse prompt label — e.g., prod-a, pointing to version 3]
- Variant B (challenger): [Langfuse prompt label — e.g., prod-b, pointing to version 4]
- Traffic weight: [e.g., 50/50 or 90/10 for gradual rollout]

SCORING CONTRACT:
- Primary metric: [e.g., task success rate scored by LLM-as-a-judge on a 0-1 scale]
- Guardrail 1: [e.g., p95 latency must not exceed 2000ms]
- Guardrail 2: [e.g., cost per request must not increase more than 15%]
- Minimum sample: [e.g., 200 requests per variant before declaring a winner]

BUILD ORDER:
1. Langfuse client: fetch both prompt versions per request, randomize using specified traffic weight, tag each trace with selected label
2. Braintrust experiment: pull sampled traces per label from Langfuse, run primary metric scorer and guardrail checks, store as immutable snapshot for comparison
3. Promptfoo CI check: run golden test set against current prompt config on every PR that touches prompt files; fail the PR if primary metric drops below [threshold]

CONSTRAINTS:
- Traffic split lives in application code — no server-side Langfuse router
- Braintrust experiments must be comparable: same input set, same scorer version, same evaluation criteria
- Promptfoo CI gate must pass before any Langfuse label is reassigned to a new version
- Langfuse SDK must be v2 or later

VALIDATION:
After build, verify:
- Each Langfuse trace carries exactly one variant label (prod-a or prod-b)
- Braintrust experiment UI shows side-by-side primary metric for both variants
- Promptfoo CI step fails the pull request when primary metric falls below [threshold] on the golden test set

Ship It

You now have three non-overlapping tool scopes, a scoring contract that predates the first routed request, and a checklist that tells you exactly what a passing experiment looks like. Prompt changes stop being guesswork when “better” is defined before the experiment starts. The next time a teammate asks whether a prompt change is ready to ship — you already have the answer written down.

Deploy safe, Max.