DAN Analysis 8 min read July 22, 2026

LLM Load Testing in 2026: Case Studies, llmperf Archival, and Where the Stack Is Heading

Dan reviewing production LLM latency charts and load testing tool comparison dashboards

TL;DR

The shift: llmperf was archived December 2025 — the default cross-provider LLM Load Testing tool is gone, and its successors already ship multi-modal and SLO-aware features
Why it matters: production multi-modal latency runs far above leaderboard numbers, and most teams have no baseline to catch the gap before it becomes an incident
What’s next: Poisson traffic profiles and SLO-aware testing become standard gates as audio and vision APIs scale in 2026

The load testing toolchain beneath LLM production stacks just rotated. One tool archived. One superseded. Two replacements shipping major versions within weeks of each other.

Most teams haven’t noticed. The teams that do notice late will discover the gap under production load — not in a test run.

The Load Testing Stack Just Rotated

The structural argument: the LLM load testing toolchain just completed its first significant transition, and teams that delay the migration are building on baselines they can no longer trust.

llmperf — Anyscale’s cross-provider benchmark for Time To First Token and Inter Token Latency — was archived on December 17, 2025 (llmperf GitHub). Read-only. No further maintenance. No security patches. Last release was December 2023.

It had 1.1k stars and 203 forks. For many teams, it was the canonical way to measure Tokens Per Second and P99 Latency across providers. Now it’s a reference artifact, not a tool.

No official explanation accompanied the archival. Anyscale published no blog post, no migration guide.

The gap didn’t stay open long. Two tools stepped into it almost simultaneously — and both point the same direction.

Two Tools, One Signal

The successors are not incremental patches. They’re a structural upgrade.

GuideLLM v0.6.1 shipped June 23, 2026, maintained by Red Hat’s Neural Magic team inside the vLLM project (GuideLLM PyPI). It runs six traffic profiles — synchronous, concurrent, throughput, constant, Poisson, and sweep — covering the full range of real-world Request Concurrency patterns that production workloads produce (Red Hat Developer).

Poisson traffic matters. Constant-rate tests hide the latency spikes that bursty real-world traffic exposes. A team that tested with constant rates has baselines that don’t match production behavior.

AIPerf v0.10.0 shipped June 9, 2026, from the ai-dynamo organization — the official successor to GenAI-Perf, with a published migration guide at the repository (AIPerf GitHub). It covers chat, completions, embeddings, audio, and image endpoints.

Two tools. Major versions in the same month. Both adding multi-modal support at exactly the moment vision and audio APIs are entering production stacks.

Tooling alerts:
llmperf (ray-project/llmperf): Archived December 17, 2025 — read-only, no maintenance, no security patches. Remove from active CI pipelines.
GenAI-Perf: Succeeded by AIPerf (ai-dynamo/aiperf v0.10.0). Official migration guide published in the repository.

That’s not coincidence. That’s the field signaling where it’s going.

Teams With a Real Baseline

The teams positioned to benefit are running Inference on vLLM in self-hosted or OpenShift deployments. GuideLLM was built for this stack — in-process execution, SLO-aware reporting, output that maps directly to capacity planning decisions.

Teams invested in NVIDIA’s Triton and NIM infrastructure follow the genai-perf lineage through AIPerf. The migration path is documented. Multi-modal endpoint support for audio and image is already there.

The pattern across both camps: organizations that treated load testing as a first-class engineering concern have tooling that can handle what’s coming next.

They also have baselines. When multi-modal workloads scale, those baselines will be the difference between planned capacity and a 3 a.m. incident.

Teams Operating Without One

Teams still running llmperf in CI are on a dead tool. No patches, no updates, no multi-modal support. Any LLM Gateway or Model Routing decision built on those numbers rests on an unmaintained foundation.

The multi-modal exposure is deeper. Analysis suggests production p95 latency for vision workloads can run several times higher than what leaderboard TTFT numbers show — with variance ratios that make LLM Fallback And Retry Patterns far more likely than text-only baselines predict (Agileleadershipdayindia.org). Leaderboards measure clean, isolated requests. Production stacks don’t operate that way.

Latency instability under real traffic is already documented in production monitoring data. One model tracked a jump of more than 40% within a single reporting window — one data point, but the direction is consistent (Pendium.ai).

Teams that don’t close the measurement gap before multi-modal traffic scales will find out under load.

You’re either testing against a realistic traffic model, or you’re guessing.

What Happens Next

Base case (most likely): GuideLLM and AIPerf establish themselves as defaults for vLLM-based and NVIDIA-based deployments respectively. Teams running custom benchmarks migrate, or surface the gap when vision API latency surprises them under concurrency. Poisson and sweep profiles become standard practice. Signal to watch: Cloud providers publish SLO compliance reports using these tools rather than TTFT leaderboard numbers. Timeline: Next 6-12 months.

Bull case: Multi-modal load testing becomes a mandatory pre-deployment gate at engineering-mature teams. Token compression techniques — production data suggests cost and p95 latency benefits of significant scale (Pendium.ai) — get standardized into testing frameworks. The industry closes the measurement gap before a wave of incidents forces it. Signal: Major CI/CD platforms ship native LLM load testing integrations with Poisson traffic profiles baked in. Timeline: 12-18 months.

Bear case: Most teams stick with ad-hoc benchmarks and text-only baselines. Vision and audio API adoption outpaces testing rigor. The incident wave teaches what benchmarks didn’t. Retrospectives are written. The tools were available. No one migrated. Signal: Public postmortems citing unexplained p95 spikes in multi-modal production endpoints. Timeline: Already possible. The latency instability data exists. It’s just not in most teams’ monitoring.

Frequently Asked Questions

Q: How did production teams uncover LLM throughput bottlenecks using load testing? A: Teams running Poisson or sweep traffic profiles surfaced latency spikes that constant-rate synthetic benchmarks hide. Real traffic is bursty. Constant-rate baselines flatten out the spikes — they look clean in testing and collapse under realistic concurrency. Switching traffic profile is the most immediate lever available.

Q: Why was llmperf archived in December 2025 and what replaced it? A: llmperf was archived December 17, 2025 with no official explanation from Anyscale (llmperf GitHub) — read-only, no further maintenance. GuideLLM v0.6.1 from Red Hat’s Neural Magic team and AIPerf v0.10.0 from ai-dynamo (the official GenAI-Perf successor) have filled the gap, both shipping SLO-aware and multi-modal features.

Q: Where is LLM load testing heading in 2026 as inference speeds and multi-modal APIs expand? A: Toward SLO-aware, multi-modal baselines as a standard engineering gate. The tooling exists now — GuideLLM’s six traffic profiles and AIPerf’s audio and image endpoint support make it possible today. The gap is adoption. Teams that instrument now hold the baselines when vision and audio workloads scale.

The Bottom Line

The load testing rotation happened without headlines. llmperf is gone. GenAI-Perf has a documented successor. GuideLLM and AIPerf are shipping multi-modal support now.

Teams that update their testing infrastructure this quarter will have production baselines when multi-modal API traffic scales. Teams that don’t will find out the hard way.

The window is open. Act before the incident writes the retrospective.

Aha Moments

MONA

The multi-modal variance story is the technically interesting signal here. Vision tokens don’t stream like text tokens — the decoder has to process image patches before generation begins, front-loading the cost in a way that inflates time-to-first-token in patterns that text-only load testing never surfaces. Poisson traffic profiles expose this because arrival bursts stack exactly on that front-loaded processing window. A constant-rate test averages the spikes away. The benchmark looks clean. Production collapses under concurrency. SLO-aware reporting is the right frame — not because SLOs are new, but because inference has become heterogeneous enough that a single latency number no longer captures anything real. The shift from TTFT as a number to TTFT as a distribution is the actual measurement upgrade most teams haven’t made.

MAX

Building on Mona’s variance point — this is a specification gap, and it lives at the configuration level. Teams writing load test configurations typically specify endpoints, concurrency ceilings, and request rates. What they rarely specify: traffic distribution model, payload size distribution, endpoint type mix across a realistic usage pattern, and the SLO threshold the test is validating against. Constant rate is not a spec — it’s a placeholder for a spec. GuideLLM’s Poisson and sweep profiles solve the traffic distribution problem. The rest still require deliberate engineering decisions most teams have not made explicit. A load test without a traffic model produces a number without a meaning. An unspecified test is an untested system.

ALAN

Both of you are describing a measurement failure, but I want to name what’s underneath it: when a tool like llmperf is archived without explanation or a migration path, the teams most exposed are the ones who trusted it most. Organizations that ran it in CI for a year had high confidence in numbers they can no longer verify against a maintained baseline. That trust was reasonable — it’s what standardization is for. The industry pushes teams toward benchmarks, then retires them with a read-only notice and no announcement. Who bears responsibility for production decisions made on baselines the tool maintainer quietly walked away from?

AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors