Gateway Proxy
Govern every LLM call with zero code changes via an OpenAI-compatible proxy.
The gateway proxy is an OpenAI-compatible HTTP endpoint that sits between your agent and the
model provider. Point your existing client at it — change only the base_url — and every call is
authenticated, policy-checked, cost-tracked, and loop-screened before it reaches the provider.
import openai
client = openai.OpenAI(
base_url="http://localhost:8000/gateway/v1", # your SteerPlane gateway
api_key="sk_sp_...", # your SteerPlane key
default_headers={"X-LLM-API-Key": "sk-..."}, # provider key, injected server-side
)
# Every request is now governed — no other code changes.
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Summarize this ticket..."}],
)What the gateway does on every request
- Authenticates the SteerPlane key and resolves the session.
- Evaluates policy (deny → allow → rate-limit → approval) before any provider call.
- Checks session cost against the configured ceiling.
- Hashes the prompt (SHA-256) for network-layer loop detection.
- Forwards to the provider, injecting the real key server-side.
- Calculates response cost and logs telemetry.
The agent process never holds the real provider key. It's stored server-side and injected
during forwarding via the X-LLM-API-Key header, so a compromised agent can't exfiltrate it.
Mid-stream cost circuit-breaker
For streaming responses (SSE), the gateway parses each chunk's token usage, accumulates the running session cost, and — the instant the ceiling is crossed — injects a termination event and severs the connection mid-stream. Unlike systems that check cost only after a call completes, overshoot is bounded to a single step.
Network-layer loop detection
The gateway maintains a per-session sliding deque of recent prompt hashes. If the same hash recurs beyond a threshold, the request is rejected before it reaches the provider — a defense-in-depth layer that works even when the SDK isn't integrated. See loop detection.