LiteLLM
Also known as: LiteLLM Proxy, LLM gateway, LLM proxy server
- LiteLLM
- LiteLLM is an open-source Python library that provides a unified OpenAI-compatible API for 100+ LLM providers. It handles model routing, cost tracking, rate limiting, and fallbacks, letting teams swap or combine AI models — including those from OpenAI, Anthropic, and Google — without rewriting application code.
LiteLLM is an open-source library that gives developers a single OpenAI-compatible API to call over 100 LLM providers, with built-in cost tracking, model routing, and fallback handling.
What It Is
Most teams that start building with AI models hard-code their provider of choice — OpenAI, Anthropic, or Google — and get things working. The problem arrives later: when costs climb, when a new model outperforms for a fraction of the price, or when a provider’s API goes down. At that point, the provider is baked into every API call in the codebase.
LiteLLM solves this before it becomes a problem. Think of it like a power strip with adapters for every country — you plug one device in, and it handles the conversion for whichever outlet you need. Write your code once using the OpenAI API shape (the de facto standard across the industry), and LiteLLM translates those calls to any provider: Anthropic, Azure OpenAI, Google Vertex, Cohere, or dozens of others.
The library ships in two modes. The first is a lightweight Python SDK you import directly into your application — no extra infrastructure, just a dependency swap that unlocks multi-provider routing from inside your existing code. The second is LiteLLM Proxy, a locally-hosted server that exposes an OpenAI-compatible endpoint. Any service that knows how to talk to OpenAI can talk to the proxy, and the proxy decides where to route the request. Teams with multiple services, or with legacy code written before multi-model routing existed, often prefer the proxy because it requires no changes in individual services.
Beyond provider translation, LiteLLM handles the mechanics that make cost management practical: automatic fallbacks when a model is rate-limited or unavailable, load balancing across providers, per-request cost tracking, and callback hooks for logging to observability platforms like Langfuse or OpenTelemetry. These features are what the cost-cutting strategies in the parent article depend on — LiteLLM is the runtime layer that makes model routing, prompt caching management, and budget enforcement accessible through a single configuration.
How It’s Used in Practice
The most common entry point is cost visibility. A team realizes they’re spending more than expected on LLM calls but can’t tell which features or models are driving the bill. LiteLLM’s built-in cost tracking logs every call with its model, token count, and calculated cost — giving the team the data they need before they start optimizing routing.
Once cost data exists, routing gets configured. A document summarization feature gets routed to a cheaper model; real-time chat stays on a faster one; a complex reasoning task stays on the highest-quality option available. All of this routing logic lives in a configuration file, not scattered across dozens of API calls. The application code stays unchanged; only the routing policy changes.
Pro Tip: Before touching routing configuration, run LiteLLM’s cost tracking for a week on your existing traffic. The data almost always reveals one or two features consuming a disproportionate share of the budget — optimize those first, rather than spreading routing complexity evenly across everything.
When to Use / When Not
| Scenario | Use | Avoid |
|---|---|---|
| Team uses multiple LLM providers and wants one interface | ✅ | |
| Automatic fallback needed when a provider is rate-limited or unavailable | ✅ | |
| Real-time per-request cost tracking across all model calls | ✅ | |
| Routing logic should live outside application code, in config | ✅ | |
| Building a single-provider MVP with no plan to switch | ❌ | |
| Existing OpenAI code works and cost is not yet a concern | ❌ |
Common Misconception
Myth: LiteLLM adds significant latency to every API call because it sits as a proxy in the middle of every request.
Reality: LiteLLM runs as a Python SDK with no server overhead at all. In proxy mode, the added latency is the round-trip to a local or internal server — typically a few milliseconds, not the hundreds you’d expect from a remote API hop. For most applications, the observability and routing benefits outweigh this cost.
One Sentence to Remember
LiteLLM is the adapter layer that lets you swap, route, or combine LLM providers without changing your application code — making it the practical foundation for the cost management strategies in the parent article on cutting LLM API costs.
FAQ
Q: Is LiteLLM free to use? A: LiteLLM is open-source and free to self-host as a library or proxy server. An enterprise tier exists for teams that need additional management features and access controls.
Q: Do I need to rewrite my existing OpenAI code to use LiteLLM? A: Minimal changes are needed. LiteLLM mirrors the OpenAI API signature, so most existing code requires only a one-line import swap and a model name update to route through LiteLLM.
Q: How does LiteLLM connect to observability tools like Langfuse? A: LiteLLM sends logs to observability platforms through callback functions. Each call automatically records the model used, token count, cost, and latency — no separate instrumentation is needed in application code.
Expert Takes
LiteLLM operates as a protocol translation layer. Different providers expose different API shapes, authentication schemes, and streaming formats. LiteLLM normalizes these into a single interface, handling the impedance mismatch between provider-specific implementations and a standardized caller signature. The cost tracking works because every normalized call passes through a metering point before and after provider dispatch — making budget accounting deterministic regardless of which backend actually processes the request.
When you configure an LLM-dependent feature, LiteLLM belongs in your architecture from day one, not as a retrofit. Define model routing and fallback logic in a configuration file that sits outside application code. When a model is deprecated or a cheaper alternative arrives, update the config — not the service. This separation of routing policy from request logic is what makes multi-model deployments maintainable without turning every model change into a deployment event.
Every team that hard-codes one provider everywhere owns a migration debt they don’t know they have yet. LiteLLM is the hedge: you keep your application code clean while the underlying model market keeps moving. When prices shift or a new model outperforms for a fraction of the cost, you’re ready to switch in a configuration file, not a codebase.
LiteLLM abstracts away model identity, and that’s worth examining. When your application routes requests automatically based on cost or speed, no one in the loop sees which model produced which answer. That’s operationally efficient and ethically opaque. Teams using LiteLLM should still log which model answered what — not for billing, but because different models have different failure modes, biases, and reliability profiles. Routing without visibility is just a faster way to be surprised.