Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
One server between your apps and every AI model provider. Apps call it in one format; it checks keys and budgets, picks a model, falls back on failure, records cost, and holds the real provider API keys.
In short
An LLM gateway (also called an AI gateway) is one server that sits between your apps and every model provider you use, such as OpenAI, Anthropic or your own GPUs. Apps send all model calls to the gateway in one format. The gateway checks who is asking, enforces budgets and rate limits, picks a model, retries or falls back when a provider fails, and records cost and logs. It also holds the real provider API keys, which makes it one of the most valuable things to protect in your system.
In short
An LLM gateway (also called an AI gateway) is one server that sits between your apps and every model provider you use, such as OpenAI, Anthropic or your own GPUs. Apps send all model calls to the gateway in one format. The gateway checks who is asking, enforces budgets and rate limits, picks a model, retries or falls back when a provider fails, and records cost and logs. It also holds the real provider API keys, which makes it one of the most valuable things to protect in your system.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers LLM Gateway across a run of lessons, not one page. These 4 alone are about 295 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Cheap models for easy traffic, strong models for hard cases. How to route, when a cascade stops paying, and the one thing your gateway must never do.
HTTP 429 is the first failure an LLM app hits at scale. Beat rate limits and provider outages with token buckets, backoff, degraded modes, and failover.
A prompt to an LLM provider is a data export. Redact PII before it reaches the model and your logs, bound retention, and honor deletion across every store.
Lesson 1's 40 messages, answered as a JSON object with three fields, asked for four ways on two models: in words, with Ollama's JSON mode, with a JSON schema, and with the schema alone. Every way gave valid JSON 40 of 40 times. The schema stopped one invalid category ('travel') but turned it into a valid wrong one; without the words, summaries grew past the limit; and with a token limit of 20, a schema did not stop the JSON from breaking.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Model Routing and Cascades: Running Three Models Without Chaospay once,
yours for life
Picture a company with five teams, each calling large language models (LLMs) from their own code. Each team has its own copy of the OpenAI key in a config file. Nobody knows which team spent what. When OpenAI has a bad hour, every app breaks in its own way. When someone leaves, nobody is sure which keys they saw.
A gateway fixes this by putting one front door in the path. Every app talks to the gateway. Only the gateway talks to the providers. That one change gives you one place to control access, one place to count money, and one place to handle failures.
The gateway usually speaks the OpenAI API format, because most client libraries already use it. LiteLLM's documentation puts it plainly: if a client can use the OpenAI API, it can use the gateway without code changes. The gateway translates each call to the format of the provider it forwards to.
Here is the path of one chat request through a typical gateway. A virtual key is a key the gateway makes for one team or app. It is not a real provider key, so it is useless anywhere except at your gateway, and you can switch it off without touching the provider.
Most gateways, open source or hosted, offer the same core list. Cloudflare's AI Gateway, for example, lists analytics, logging, caching, rate limiting, and request retry and model fallback.
It is the same idea as an API gateway in front of microservices, with features shaped for model calls: tokens instead of requests as the unit of cost, and provider fallback as a normal path rather than an emergency.
A gateway that holds every provider key, sees every prompt and stores every log is exactly what an attacker wants. On 21 October 2026 this stopped being a theory.
A poisoned LiteLLM release, version 1.107.0, was on PyPI (the Python package index) from 11:39 to 13:23 UTC, according to LiteLLM's incident post. Docker images under the v1.107.0, latest, main-latest and main-stable tags were affected for a similar window. The package added a file that runs every time Python starts, even if your code never imports litellm. Among many other things, it read the database URL and dumped the tables in which a LiteLLM proxy keeps its provider API keys. Version 1.106.2 is clean.
The maintainers' own updates add a second lesson. They say the attacker most likely took GitHub login tokens from the logs of their own internet-facing LiteLLM deployment: a coding tool had put the tokens into response bodies, and the logs kept those bodies. The gateway's logs became the way in.
So treat a gateway like a secrets vault, not like a convenience proxy.
"Design an LLM gateway for a company with 50 teams" is now a common AI system design question, and the gateway also shows up inside bigger designs such as a support agent or a RAG system. A strong answer covers the request path above, how virtual keys and budgets are stored (a database, with a cache in front so the hot path does not hit the database on every call), how fallback avoids sending the same failing request twice in a loop, and how to count tokens for streaming responses, where the total is only known at the end.
The answer that stands out also covers the security side: where the provider keys live, what is logged and for how long, and how the gateway itself is built and deployed. After this week, interviewers will ask.
The AI Engineering course covers this end to end, from routing between models to tracing every call.
LiteLLM
An open-source, self-hosted gateway with one OpenAI-compatible endpoint for 100+ providers, virtual keys with budgets, fallbacks and an admin UI. Version 1.107.0 was a poisoned release on 21 Oct 2026; 1.106.2 is clean.
Cloudflare AI Gateway
A hosted gateway that adds analytics, logging, caching, rate limiting, retries and model fallback with one line of configuration.
OpenAI
The API format most gateways copy, so existing client code works unchanged when pointed at the gateway.
Anthropic
A typical second provider behind a gateway, used as a fallback when the first provider returns errors or rate limits.
It is one server that every app in your company uses to call AI models. The apps send requests to it, and it forwards them to OpenAI, Anthropic or your own models. On the way it checks who is asking, enforces budgets, retries or switches provider on failure, and records the cost.
Yes, the two names are used for the same thing. Some vendors say AI gateway because the same server can also front tools and agents, not only language models.
Both are a single front door. An API gateway routes requests to your own services. An LLM gateway routes model calls to outside providers, counts tokens and cost, and falls back between providers as a normal part of its job.
Usually not. A gateway pays off when several teams or apps share providers, when you need spending limits per team, or when you want to switch or fall back between providers without changing every app.
It can be, if you treat it like a secrets vault: keep it on a private network, keep provider keys in a secret manager, redact or skip storing request bodies, pin and verify the exact version you install, and watch its outbound traffic.
API Gateway
A single entry point for all client requests that routes them to the appropriate microservice. Handles auth, rate limiting, and request transformation.
Rate Limiting
Controlling how many requests a client can make in a given time window. Protects your API from abuse and ensures fair usage.
API Key
A simple token passed with API requests to identify the calling project or application. Not a substitute for user authentication but useful for rate limiting and usage tracking.
Secret Management
Securely storing and distributing credentials, API keys, and certificates. Tools like Vault, AWS Secrets Manager, and SOPS prevent secrets from leaking into code or logs.
LLM Observability
Recording what an LLM application actually did on each request, the prompts, model calls, tool calls, tokens, cost and latency, so you can explain and fix a bad answer.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
This page explains the idea. The "Model Routing and Cascades: Running Three Models Without Chaos" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Model Routing and Cascades: Running Three Models Without Chaos lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about LLM Gateway in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
