Surge AI, Scale AI, and the LLM-as-Judge Shift: Human Evaluation in 2026

The deal wasn’t the news. The exits were.

Meta acquired a 49% nonvoting stake in Scale AI for $14.3 billion, valuing the company at $29 billion (TechCrunch). Alexandr Wang stepped out as CEO to become Chief AI Officer at Meta Superintelligence Labs. That landed on June 13, 2025. Five days later, OpenAI announced it was dropping Scale AI as a data provider (TechCrunch). Google — which had planned roughly $200 million in Scale contracts for the year — followed. Microsoft and xAI went the same direction.

Every lab competing with Meta looked at its primary evaluation vendor and saw a conflict it could not live with.

The Conflict Map Just Became Official

Thesis: Meta’s stake turned Scale AI from neutral infrastructure into a Meta affiliate — and neutrality is the only thing that makes an evaluation vendor worth using.

Human Evaluation for AI sits at the core of frontier model training. Expert labelers score outputs, surface failure modes, and shape preference data. The better your evaluation partner, the better your model gets — and the more your competitor knows about what you are optimizing against.

Scale’s 2024 revenue ran at $870 million. Google alone accounted for roughly $150 million of that (TechCrunch). That is gone now. So is the business from OpenAI, Microsoft, and xAI. Jason Droege, Scale’s interim CEO, inherited a company with a new parent that happens to be the primary competitor of its former biggest clients.

In November 2025, Scale’s CFO told CNBC the company was “not a zombie.” The defense was noted.

The Market Reshuffled Within Weeks

The fast movers were immediately visible. Appen, Turing, Labelbox, Mercor, and Handshake all reported client inflows after the Meta deal closed (Computerworld).

The most telling signal came from Surge AI. Surge had spent years running the RLHF platform for Anthropic’s Claude — domain expert labelers across math, coding, law, and medicine, plus red teaming and quality control — while simultaneously building AdvancedIF for Meta, a human-expert benchmark that delivered measurable instruction-following gains (Surge AI Blog). Both clients. Full independence holding both relationships open.

Surge entered fundraising talks mid-2025, reportedly targeting a valuation between $15 billion and $25 billion — its first capital raise after bootstrapping to over $1 billion in annual revenue with roughly 50,000 expert contractors (SiliconANGLE). The raise had not publicly closed as of this writing.

Meanwhile, the LLM as a Judge category accelerated. Atla’s Selene 1, built on Llama-3.3-70B, outperformed GPT-4o and Claude 3.5 Sonnet across eleven evaluation benchmarks (Atla AI Blog). The 8B Selene Mini became the strongest small judge model available. LLM-judge agreement with human evaluators runs around 85% on average — but Cohen's Kappa scores vary from 0.17 to 0.86 by domain (arXiv). On nuanced, high-stakes labeling tasks, that spread matters. The research consensus: hybrid systems, not wholesale replacement.

Who Positions for the Split

Surge AI holds the clearest structural advantage. Serving Anthropic and Meta simultaneously while remaining independent — that combination is now the template for what an evaluation vendor must be in this market.

SWE Bench and ELO Rating already anchor competitive model rankings. Judge LLMs that can calibrate those scores reliably move from optional to essential infrastructure. Providers like Atla sit at exactly that intersection.

Mid-tier vendors that absorbed Scale’s displaced client contracts gain near-term revenue. Whether that converts to durable market position depends on whether they can match the domain-expert quality bar Surge established.

Who Takes the Penalty

Scale AI absorbed the cost of being the wrong choice for most of the market simultaneously.

The CFO needed to argue “not a zombie” on national business television. That phrase is a concession, not a position.

Data vendors running concentrated client books learned Scale’s lesson at one remove. One corporate transaction, and structurally latent risk becomes operational reality overnight.

You’re either building evaluation infrastructure with diversified, independent vendor relationships or you’re one M&A event away from rebuilding your data pipeline from scratch.

What Happens Next

Base case (most likely): Scale AI stabilizes around its applications business and a smaller diversified labeling roster. The era of Scale as neutral mega-vendor is over. Surge closes its raise and becomes the independence standard-bearer in human evaluation. Signal to watch: Scale’s 2026 revenue figures or a major new labeling contract announcement. Timeline: 12–18 months.

Bull case: LLM-as-judge improves fast enough that human labeling volume compresses broadly by mid-2027. Winners are hybrid platforms owning both automated and human evaluation stacks. Signal: Enterprise adoption of specialized judge LLMs for production RLHF pipelines at scale. Timeline: 18–24 months.

Bear case: A second major M&A event concentrates another market segment. Surge raises capital from a frontier lab and the independence premium disappears. Signal: Surge AI announces a strategic investment from one of the major AI labs. Timeline: 12 months.

Frequently Asked Questions

Q: How does Surge AI run human evaluation for Anthropic’s Claude? A: Surge provides domain expert contractors — mathematicians, lawyers, coders, doctors — who score Claude’s outputs against structured Rubric Design criteria. The pipeline includes red teaming, quality control layers, and rapid experimentation APIs for preference labeling at scale (Surge AI Blog).

Q: Why did Scale AI clients diversify after Meta’s $14.3B investment in 2025? A: Meta’s 49% stake made Scale AI a de facto Meta affiliate. Labs competing with Meta — Google, OpenAI, Microsoft, xAI — faced an unacceptable conflict: their training data and evaluation pipelines flowing through a vendor now partially owned by their primary competitor (TechCrunch).

Q: Will LLM-as-judge make human evaluation obsolete by 2026? A: No. Automated LLM-as-judge evaluation reaches strong average agreement with human raters, but reliability varies significantly by task domain. Expert human evaluation remains essential for safety-critical labeling and edge cases. The direction is hybrid — Label Studio pipelines augmented by judge models, not replaced by them (arXiv).

The Bottom Line

The Scale AI–Meta deal made explicit what was always structurally true: neutrality is the product, not the service. Surge AI built that. Scale AI lost it. The market is voting accordingly.

The winners in this market are independent or they are nothing.