What is Manifest?
Manifest is an open-source LLM gateway for builders and teams running AI agents, applications, and workflows. It routes requests across subscription plans, pay-per-token providers, custom endpoints, and local models, with options to manage budgets and inspect inference usage. Use it in the cloud or run it on your own infrastructure, and connect compatible agents or harnesses through the gateway.
Manifest Features:
Request Autofix: Manifest repairs supported broken LLM requests before returning them to the agent, and the Autofix capability is also available as a standalone option.
Provider Fallbacks: Set up alternate providers or models so requests can be retried when a provider becomes unavailable.
Flexible Routing: Define routing rules and model parameters for each route, choosing among subscription services, API-key providers, compatible custom endpoints, and local models.
Usage Logs: Inspect the full request and response body for each call, and review past queries for workflow optimization.
Spend Controls: Track costs by agent, key, and provider, and set per-key daily, weekly, or monthly caps with alerts.
Self-Hosting: Run the gateway on your own machine or infrastructure using Docker, with support for local models including Ollama, LM Studio, and llama.cpp.
Manifest Benefits:
Fewer Disruptions: Request repair and provider retries can help keep agents and workflows moving when calls fail or a provider goes down.
Provider Choice: Combining subscriptions, usage-based services, and local models gives builders more ways to match inference choices to each request.
Clearer Spending: Cost breakdowns and per-key caps make it easier to see where inference spending comes from and respond before a limit is reached.
More Data Control: Self-hosting lets teams keep the gateway and its data on their own infrastructure rather than sending data to a hosted gateway.
Less Manual Tuning: Visual route and parameter settings let teams adjust model choices without changing application code for each configuration.
Use Cases:
Personal Agents: Connect a personal agent to paid AI subscriptions, local models, or usage-based APIs, and set spending limits for its calls.
Team Cost Reviews: Give engineers separate keys and use spend breakdowns to identify which agents, people, or providers account for team usage.
AI Application Routing: Choose different providers or models for application queries, with fallback routes available when a provider is unavailable.
Local Model Workflows: Route selected workflow requests to models running through Ollama, LM Studio, llama.cpp, or another local server.
Workflow Optimization: Review and replay past queries while tuning model and provider choices for an AI workflow.