What is sturnus?
sturnus is a lightweight sidecar that routes requests across configured LLM providers according to their observed latency and success rates. It exposes an OpenAI-compatible API, so applications using compatible SDKs can send requests through it by changing the API base URL. Teams define model aliases and interchangeable provider-model candidates in a TOML configuration; sturnus then directs traffic toward the fastest healthy option it observes, while running inside the team’s own infrastructure.
sturnus Features:
Latency-Aware Routing: sturnus measures time to first chunk and success rate for each candidate, then weights traffic toward the options with the lowest effective latency.
Model Aliases: A TOML file maps client-facing model names to interchangeable provider and model pairs, letting the sidecar route requests without changing the alias clients send.
Session Affinity: An x-session-affinity response header can be sent with follow-up requests to keep them on the same provider, unless that candidate’s error rate breaches the configured threshold.
OpenAI Compatibility: The sidecar accepts OpenAI-compatible requests for chat completions and embeddings, rewrites the model field, and relays responses, including server-sent event streams.
Bounded Buffers: Request buffers have per-request and aggregate limits, and requests that exceed the available memory budget receive a 429 response with a Retry-After header.
Metrics and Status: The service provides health and status endpoints plus Prometheus metrics for requests, response timing, transport failures, and buffer rejections.
sturnus Benefits:
Less Waiting: When equivalent model options perform differently, routing toward the faster healthy candidates can reduce the time users spend waiting for responses.
Automatic Recovery: Providers retain a small share of traffic for measurement, so a recovered or newly responsive option can earn traffic again without manual switching.
Fewer Code Changes: Teams can direct compatible clients through the sidecar by changing the base URL, avoiding changes to the request flow for provider selection.
Consistent Sessions: Keeping follow-up requests with the same provider can help preserve continuity when a multi-turn conversation depends on provider-specific context.
Local Operation: Running a small sidecar inside existing infrastructure avoids depending on a hosted routing service or shared database for routing state.
Use Cases:
Production LLM Traffic: Route an application’s requests across interchangeable provider-model options when upstream response speed and availability vary over time.
Multi-Turn Chat: Pass the affinity header between turns of a conversation when follow-up requests should stay with the provider that handled the earlier request.
Kubernetes Deployment: Run sturnus as a native sidecar bound to loopback, with each instance routing from the performance it observes locally.
Provider Monitoring: Expose Prometheus metrics and status information when operators need to inspect request counts, timing, errors, and candidate health.
Vertex AI Workloads: Route requests to Vertex AI from GKE using Workload Identity authentication and automatic token refresh, without configuring an API key for that provider.

