What is OpenLIT?
OpenLIT is an open-source platform for observing and improving the systems around AI agents and coding agents. It brings together OpenTelemetry-based tracing, evaluations, guardrails, prompt management, and cost and GPU monitoring, and can be self-hosted under the Apache 2.0 licence. It is for teams that already have an agent or AI application and want to understand its behavior, assess its outputs, and refine how it works in production.
OpenLIT Features:
LLM Tracing: Trace LLM calls, tool calls, retrieval, and agent steps using OpenTelemetry, with filters for fields such as user, session, cost, latency, and metadata.
Output Evaluations: Evaluate production traffic or experiments with LLM-as-a-judge, heuristic, and human-review methods, including checks for hallucination, toxicity, relevance, and instruction following.
Prompt Management: Store and version prompts in Prompt Hub, retrieve them through an API, and deploy or roll back changes without shipping application code.
Runtime Guardrails: Apply built-in checks for prompt injection, jailbreaks, sensitive topics, and topic restrictions through SDK-based guards and rule-engine automation.
Coding Agent Monitoring: Inspect coding-agent sessions, including prompts, model and tool activity, token usage, cost, repository activity, and code impact.
Cost and GPU Metrics: Track AI workload cost and latency, and collect GPU utilization, memory, temperature, and power data during inference.
Flexible Instrumentation: Send telemetry through OpenLIT SDKs, zero-code eBPF instrumentation, a GPU collector, or existing OpenTelemetry exporters.
OpenLIT Benefits:
Faster Debugging: A trace that connects model calls, tools, retrieval, and errors helps teams narrow down where a faulty agent request went wrong.
Safer Releases: Evaluating real requests and experiments gives teams a way to check quality and safety before changing an agent or its prompts.
Lower Operating Costs: Visibility into spend, latency, and hardware use helps teams identify costly or slow parts of an AI workload.
Prompt Changes With Less Friction: Managing versions and rollbacks outside application releases makes it easier to adjust prompts without rebuilding and redeploying the app.
Data Control: Self-hosting keeps the platform, traces, prompts, and secrets in the team's own environment rather than requiring a hosted service.
Use Cases:
Production Agent Debugging: Use traces to investigate a customer-facing agent request that failed after an LLM call, tool action, or retrieval step.
Answer Quality Checks: Run evaluations on live traffic or test experiments to review an agent's relevance, completeness, safety, or instruction following.
Prompt Iteration: Manage prompt versions and roll back a change when a new instruction produces worse results in an existing application.
Coding Agent Review: Review a Claude Code, Cursor, or Codex session to understand its tool use, token consumption, repository activity, and code impact.
Inference Cost Review: Compare workload cost and latency with GPU utilization and memory when investigating the resources used by LLM inference.