API Gateway

An API gateway is a server that sits between clients and backend services, routing requests, enforcing authentication, and applying rate limits in a single layer before traffic reaches any backend.

What It Is

When you call multiple services from a single application — a payment processor, a user database, an AI model API — each one has its own authentication scheme, error format, and rate limit policy. Managing that directly from client code means duplicating logic everywhere and exposing credentials to more surfaces than necessary.

An API gateway solves this by acting as the single front door. All traffic enters through it, and it decides where each request goes, whether the caller is authorized, whether the call falls within rate limits, and what gets logged. Think of it as the security desk and routing switchboard for a large office building: visitors check in once, get directed to the right floor, and security keeps a log of every entry.

The gateway pattern has three core jobs. Routing maps incoming requests to the correct backend service — a request to /chat goes to one endpoint, /embeddings to another, and /image-generation to a third. Policy enforcement checks authentication tokens, applies per-client rate limits, validates request schemas, and can rewrite headers before forwarding. Observability centralizes logging, metrics, and tracing so you see traffic patterns in one place instead of instrumenting every individual service.

In the context of LLM infrastructure, this pattern has been adapted into what practitioners call an LLM gateway. Where a general API gateway routes traffic between microservices, an LLM gateway handles the specific concerns of AI model calls: managing provider API keys under a single virtual key, applying cost budgets per team, and routing requests between models based on latency or availability — including fallback rules that trigger when a primary model goes down or exceeds rate limits.

Understanding the general gateway concept first makes LLM-specific routing and unified auth easier to reason about. The rules that govern which model gets called, what credentials are used, and what gets logged all live at this layer — not scattered across every service that makes model requests.

How It’s Used in Practice

The most common entry point for developers working with AI services is a gateway that sits in front of one or more model provider APIs. A team wants to call one provider for their main application but fall back to another during outages. Instead of building retry logic into every service that makes model calls, they point all requests at the gateway. The gateway holds the provider credentials, applies the fallback rule, and forwards traffic — individual services only need one endpoint and one authentication token.

This setup also enables cost tracking without modifying application code. The gateway logs which team or user triggered each request, which model responded, and how many tokens were used. Finance gets a report; engineering does not have to instrument every call.

Pro Tip: If you’re evaluating whether to add a gateway, start by listing which routing and auth decisions your application currently makes in code. A gateway pays off when those decisions need to change frequently — switching providers, adjusting rate limits per user tier, adding a new model to the rotation — and when more than two services are making model calls. For a single service hitting one provider, the added network hop rarely earns its keep.

When to Use / When Not

Common Misconception

Myth: An API gateway is just a reverse proxy with a marketing name.

Reality: A reverse proxy forwards requests to a backend — that is one of the things a gateway does. But a gateway adds policy enforcement on top: authentication, rate limiting, quota management, request and response transformation, and centralized logging. The distinction matters in practice: a reverse proxy alone gives you no auth layer, no per-client quotas, and no observability without additional tooling.

One Sentence to Remember

An API gateway is the centralized layer that handles the decisions — who can call, how often, and where the request goes — so individual services do not have to make those decisions themselves.

FAQ

Q: What is the difference between an API gateway and an LLM gateway?

A: An API gateway routes general service traffic and enforces policy. An LLM gateway is a specialized variant that handles AI-specific concerns: managing provider keys, routing between models, applying token budgets, and executing fallback strategies when a model is unavailable.

Q: Does adding a gateway introduce latency?

A: Yes, by single-digit milliseconds for most requests. For AI model calls that take hundreds of milliseconds or more, this overhead is negligible. For sub-10ms latency requirements, measure the actual impact before committing to the pattern.

Q: Can I use an existing API gateway for LLM routing, or do I need a specialized one?

A: General-purpose gateways handle auth and routing but miss LLM-specific features: virtual key management, token-based rate limiting, and provider fallback. A general gateway works for simple AI usage. A purpose-built LLM gateway is easier to configure for multi-provider setups.