MAX guide 14 min read July 30, 2026

How to Build and Evaluate a Domain-Specific Prompt Pipeline with LangChain, Braintrust, and Promptfoo in 2026

Domain-specific prompt pipeline showing role context, knowledge injection, and Promptfoo evaluation for enterprise AI

TL;DR

Domain-Specific Prompting requires explicit context architecture — better wording alone won’t fix compliance failures
Write your evaluation contract before you build anything; what “correct” means must be specified, not discovered after the first production incident
Three tools, three roles: LangChain routes context, Braintrust scores outputs, Promptfoo automates regression

Here’s how it usually goes. The financial prompt passes every manual review. QA signs off. You ship it. Two weeks later, the risk team flags a batch of responses for compliance issues nobody tested — because nobody wrote down what “compliant” meant before you built the pipeline. Not a model problem. Not a tooling problem. A specification problem.

Before You Start

You’ll need:

Claude Code, Cursor, or Codex as your AI coding tool
LangChain 1.3.11 (Python 3.10–3.14) — see the compatibility note in Step 3 before installing
A Braintrust account (Starter tier is free)
Promptfoo 0.121.17 (Node.js 20.20.0 or 22.22.0+)
Working knowledge of System Prompts, Role Prompting, and Prompt Engineering

This guide teaches you: how to decompose a domain-specific prompt pipeline into three independently specifiable layers — context injection, evaluation contract, and regression testing — so that your AI tool can build each layer correctly and you can catch failures before they reach production.

When Generic Prompts Break in Regulated Production

The failure mode is always the same shape. You write a prompt that handles your test cases. It doesn’t specify the regulatory frame, the output constraints, or what an incorrect-but-plausible response looks like. The model fills in the gaps — confidently, quickly, and sometimes with exactly the kind of output that would fail an audit.

It ran clean on Friday. Monday morning, a different document type hit the pipeline. The Context Window filled with irrelevant carry-over from the previous session. Nobody specified that multi-turn state needed clearing between requests.

The root cause is almost always underspecification — not the model, not the tools, not the data.

Step 1: Map Your Domain Context Layers

Before you write a single chain, map what your pipeline actually needs to inject per request. Most teams treat context as a single blob. That’s why they can’t debug it.

Domain context has three distinct layers. Separate them now, before your AI tool touches a line of code.

Layer 1 — Role context: Who is the AI in this domain? A senior risk analyst? A compliance reviewer? A financial modeling assistant? This is your Role Prompting layer. It sets professional frame, tone, and epistemic standard. It goes in the system prompt. It does not change per request.

Layer 2 — Knowledge context: What does the AI need to know for this specific request? Regulation text, financial definitions, product constraints, portfolio limits. This is your Knowledge Injection layer. It changes per request. It does not belong in the system prompt.

Layer 3 — State context: What happened in prior turns? Which documents are already in scope? What decisions were already made? This is your Multi-Turn Prompt Design layer. It must be managed explicitly — the model has no memory by default.

Your system has these parts:

Role layer — static, defines professional authority and epistemic constraints; updated only when the role changes
Knowledge layer — dynamic per request, sourced from retrieval or structured lookup; versioned to your document corpus
State layer — dynamic per session, managed explicitly with session identifiers; cleared on session end

The Architect’s Rule: If you mix these three layers into one prompt block, you lose the ability to debug which signal caused the failure. Separate layers give you separate dials.

Step 2: Write the Evaluation Contract Before You Code

Here is where most domain-specific pipelines skip a step. They build first, then figure out what “good” means by looking at outputs. Backward. Define the evaluation contract first — then every build decision follows from it.

An evaluation contract answers three questions before you generate anything:

What does a correct answer look like? (Format, structure, referenced sources, confidence markers)
What does a compliant answer look like? (Which regulations it must acknowledge, what it must not assert)
What does a hallucinated answer look like for your domain? (Invented citations, fabricated figures, regulatory references that don’t exist in the injected context)

Evaluation checklist:

Output format specified — JSON, structured prose, or table; pick one, document it, enforce it
Regulatory frame named — jurisdiction, regulation version, applicable date
Citation behavior defined — output must cite source document; must not reference documents absent from the knowledge context
Numerical constraints set — acceptable value ranges, units, uncertainty handling
Refusal behavior specified — what the model returns when the request falls outside defined scope

The Spec Test: If your evaluation contract doesn’t produce a clear pass/fail on a borderline response, the contract is not complete. Ambiguity in the rubric becomes a production bug.

Step 3: Wire LangChain, Braintrust, and Promptfoo

With your context layers mapped and your evaluation contract written, you have enough to specify the pipeline to your AI coding tool. The three tools divide responsibility cleanly.

LangChain 1.3.11 handles chain construction — stitching together your retrieval step, role context injection, and generation call. Use dedicated integration packages — langchain-openai, langchain-anthropic, or other provider-specific packages. The langchain-community package was archived June 19, 2026 (LangChain GitHub). It is now read-only, receives no further updates, and any import from it will fail without migration.

LangSmith (LangChain’s tracing layer) shows you exactly what entered each prompt. The Developer plan is free — 5,000 traces per month, 14-day retention, enough to debug a staging pipeline without a budget conversation (LangChain’s pricing page). When a prompt behaves unexpectedly, the trace isolates which layer was responsible.

Braintrust sits at the evaluation layer. You push pipeline outputs to Braintrust’s scoring API and run them against the rubric you defined in Step 2. The Starter plan is free — $10 in credits, 10,000 scores per month. For high-volume production evaluation, Pro runs $249/month (Braintrust’s pricing page).

Promptfoo 0.121.17 closes the regression loop. You write assertions against your evaluation contract — structural checks, semantic similarity, domain-specific validation functions — and run them automatically on every prompt change. OpenAI acquired Promptfoo in March 2026 (OpenAI Blog). The tool remains open source under MIT. Its roadmap points toward the OpenAI Frontier platform, so verify standalone availability for your deployment context before committing it as core infrastructure.

Context specification for your AI tool:

Tech stack: LangChain 1.3.11, Python 3.10+
Integration packages: langchain-openai or langchain-anthropic (not langchain-community)
Evaluation target: JSON output matching your contract schema
Tracing: LangSmith Developer (5,000 traces/month, 14-day retention)
Assertion runner: Promptfoo (Node.js 20.20.0 or 22.22.0+)

The same three-layer architecture extends naturally to Multimodal Prompting workloads — when your pipeline needs to process charts, scanned regulatory documents, or tables alongside text, the knowledge injection layer grows to include image context and the evaluation contract adds visual-output criteria. The pattern stays the same; the layer contents change.

Security & compatibility notes:
langchain-community archived (June 19, 2026): Read-only, no further updates. Migrate all imports to dedicated packages (langchain-openai, langchain-anthropic, etc.). Code importing from langchain-community will fail without migration (LangChain GitHub).
Braintrust API key security: Braintrust disclosed unauthorized access to their AWS account containing customer API keys on May 5, 2026 (TechCrunch). No evidence of broader data exposure was reported; all customers were instructed to rotate keys. For security-sensitive pipelines handling regulated financial data, run evaluation traces in an isolated environment and rotate Braintrust API keys on a schedule.
Promptfoo post-acquisition: Acquired by OpenAI (March 2026). Remains MIT open source. Integration into OpenAI Frontier is planned — future as an independent CLI is not guaranteed. Evaluate fit before committing as infrastructure.

Step 4: Assert Against Domain Ground Truth

Validation here is not “run a few prompts and check if they look right.” That approach tops out at maybe twenty test cases before attention degrades. Assertions don’t degrade.

Validation checklist:

Structural assertion: output matches the JSON schema from your evaluation contract — failure looks like missing required fields, wrong data types, unstructured prose where structured output was specified
Citation assertion: every source reference in the output appears in the injected knowledge context — failure looks like model citing documents not in the prompt, or inventing regulatory references
Numerical range assertion: output figures fall within specified bounds — failure looks like values outside the defined range without an explicit uncertainty flag
Refusal coverage: send out-of-scope requests and verify the model declines correctly — failure looks like a confident response on a question outside the defined domain

Promptfoo handles the first three automatically once you’ve written the contracts. The fourth — refusal coverage — needs a negative test set: requests the pipeline should explicitly refuse. Build that test set during Step 2, alongside your positive examples. Don’t wait for the first production incident.

Three-layer domain context architecture (role, knowledge, state) feeding a generation step, with Braintrust scoring outputs and Promptfoo running regression assertions in parallel — A domain-specific prompt pipeline separates context into three independently specifiable layers, each with its own evaluation criteria and failure signature.

Common Pitfalls

What You Did	Why AI Failed	The Fix
Mixed role and knowledge context into one system prompt	Can’t isolate which signal caused the wrong output	Separate into role layer (static) and knowledge layer (dynamic per request)
Skipped the evaluation contract	“Correct” was defined retroactively by what looked good to the reviewer	Write the rubric before writing any prompts
Imported from `langchain-community`	Package archived June 2026 — all imports now fail	Migrate to `langchain-openai`, `langchain-anthropic`, or the relevant dedicated package
Used only manual spot-checks	Eye-review degrades after twenty cases; regressions slip through	Automate assertions with Promptfoo before the pipeline goes to production
No negative test set	Pipeline was never tested on out-of-scope requests	Build refusal coverage alongside positive examples in Step 4

Pro Tip

The evaluation contract you write for this pipeline works for your next one. The domain changes — credit risk today, compliance checking tomorrow. The structure stays the same: role layer, knowledge layer, evaluation rubric, assertion suite. Every pipeline you build on this scaffold takes less time to specify and fails in fewer ways. That compounding is the real return on writing the contract before you code.

Frequently Asked Questions

Q: How to build a domain-specific prompt pipeline step by step with LangChain and Braintrust in 2026?

A: The four steps here — map context layers, write the evaluation contract, wire the stack, add assertions — are the minimum viable specification. One edge case worth knowing: if you’re running LangSmith tracing on langchain-core below version 1.2.4, you’ll lose token count and cost metrics in your traces. Upgrade to langchain-core 1.4.8 to keep full tracing visibility across all pipeline stages.

Q: Can domain-specific prompting replace fine-tuning for regulated enterprise AI applications in 2026?

A: For most use cases, the 2026 consensus is hybrid: domain prompting plus retrieval handles the majority of cases. Fine-tuning remains the better choice for compliance-critical scenarios where EU Act documentation requirements make the audit trail for a fine-tuned model cleaner than a complex prompt chain. If your compliance team needs to explain model behavior to a regulator, a fine-tuned artifact is a simpler story than a layered prompt architecture.

Q: How to use domain-specific prompting for financial modeling, risk scoring, and compliance checking?

A: Ground your chain-of-thought in expert financial reasoning at the knowledge injection layer — the FinCoT pattern from a June 2026 arXiv preprint shows that step-by-step reasoning anchored to retrieved documents significantly improves accuracy for financial Q&A (arXiv FinCoT). For compliance checking specifically, make regulatory references explicit in injected context and assert in Promptfoo that output citations match injected sources — not just that output cites something.

Your Spec Artifact

By the end of this guide, you should have:

A three-layer context map documenting each layer’s source, update frequency, and storage mechanism (role layer: static system prompt; knowledge layer: retrieval index; state layer: session store with explicit clear triggers)
An evaluation contract with four rubric dimensions: structural correctness, citation behavior, numerical bounds, and refusal coverage — each with a clear pass/fail criterion and a concrete failure signature
A Promptfoo assertion suite with at least one automated test per rubric dimension, including a negative test set covering the five most likely out-of-scope request types for your domain

Your Implementation Prompt

Copy this into Claude Code, Cursor, or Codex. Fill in the bracketed values with your domain specifics before sending.

Build a domain-specific prompt pipeline for [your domain: financial modeling / risk scoring / compliance checking].

CONTEXT ARCHITECTURE:
- Role layer (system prompt, static): [professional role and epistemic authority — e.g., "senior credit risk analyst with expertise in Basel III capital requirements"]
- Knowledge layer (injected per request): [retrieval source — e.g., "regulatory PDFs chunked at [token size], indexed in [vector store]"]
- State layer (managed per session): [what persists across turns — e.g., "current portfolio context, decisions made this session; cleared on session end"]

EVALUATION CONTRACT:
- Output format: [JSON schema / structured prose / table — choose one]
- Citation requirement: each response must cite at least one document from the knowledge context; citations must not reference documents absent from the injected context
- Numerical constraints: [output values must be in range X to Y, or must flag uncertainty explicitly]
- Refusal behavior: model must decline requests outside [defined scope], returning [specified decline message]

BUILD ORDER:
1. Role layer injection — static system prompt, no dynamic variables yet
2. Knowledge retrieval and injection — document retrieval, chunking, context assembly
3. Generation chain — combine role + knowledge + request
4. Evaluation wiring — Braintrust scoring API, Promptfoo assertion suite

VALIDATION:
- Structural assertion: output matches schema (no missing fields, correct types)
- Citation assertion: all cited documents present in injected context
- Numerical assertion: all figures within specified bounds
- Refusal coverage: [N] negative test cases for out-of-scope request types

STACK:
- LangChain 1.3.11, Python 3.10+
- Integration: [langchain-openai / langchain-anthropic] — do NOT import from langchain-community (archived June 2026)
- Tracing: LangSmith Developer (5,000 traces/month)
- Assertions: Promptfoo 0.121.17, Node.js 20.20.0+

Ship It

You started with a prompt and a hope that it would work in production. You now have a three-layer context architecture, an evaluation contract with four rubric dimensions, and an assertion suite that runs on every change. The model hasn’t changed. What changed is that you finally told it — and yourself — exactly what “done” looks like. Deploy safe, Max.

Aha Moments

MONA

The separation Max describes here maps to something structural, not just organizational. When you isolate role context, knowledge context, and state context into separate layers, you are creating distinct input signals that the model processes in sequence. Without those boundaries, the model merges them during attention computation — and which signal dominates cannot be predicted reliably from outside the model. The LangChain architecture is an editorial contract: this goes in the system prompt, this goes in the retrieval layer, this gets injected per turn. The model does not care about your intentions. It sees token sequences. This layered context architecture is how you translate intention into token structure — and that is not a design metaphor, it is the mechanism that determines what the model attends to first.

DAN

Mona is right that the model doesn’t care about your intentions — but the compliance officer does. That is the business problem this pipeline actually solves. What Max has built here is an evaluation contract that translates technical correctness into something a regulatory audit can read. You are not just checking whether the model answered correctly. You are checking whether the answer would survive scrutiny from someone who has never seen your codebase. That translation — from Promptfoo assertion to audit trail — is exactly where most enterprise AI programs collapse. The tools are available. The evaluation discipline is not. Braintrust and Promptfoo give you the infrastructure. The hard part is writing the rubric that your compliance team and your engineering team can both sign off on without a two-week debate.

ALAN

Dan has the right framing, but it leaves something out. The pipeline Max describes is excellent at catching failures you anticipated when writing the evaluation rubric. It will not catch failure modes you did not think to test for. In a financial context, the most dangerous outputs are the ones that pass all current assertions but would fail a human expert review for reasons not yet in the rubric. Every pipeline has a blind spot shaped exactly like the rubric that defines it. The security incident already embedded in this article — the kind where evaluation infrastructure itself becomes an attack surface — is an argument for treating your Braintrust traces and Promptfoo assertion data with the same access controls as your model outputs. The question this guide leaves open: who writes the rubric for the rubric, and on what authority?

AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors