What is LLMxRay?
LLMxRay is a local observability interface and proxy for Ollama-based language models. It sits between Ollama and compatible clients, forwarding requests while recording per-turn metrics and showing how prompts, cache reuse, model loading, and tool calls behave. Developers and teams running local models can inspect individual requests, review agent runs, and investigate slowdowns or unexpected prompt changes. It also includes a chat interface for trying a model directly, along with views for GPU memory and agent tool-call behavior.
LLMxRay Features:
Live Request Monitoring: A Live page displays each client request with the model, prompt size, cache use, prefill and generation speed, and any tool calls.
Prompt X-Ray: Prompt bars show system instructions, tool schemas, earlier turns, and new messages, highlighting cached content and text cut by the daemon.
Run Waterfall: A timeline separates model waiting, loading, prompt processing, and generation from the time between calls when an agent runs tools.
Shared Box View: A per-tool view sorts requests by their impact on other users and shows settings that can cause model unloading or queue delays.
GPU Memory Map: A GPU view breaks loaded models into weights and context usage, then indicates which pulled models fit in available memory.
Cache Lab: Cache Lab finds prompt values that change between turns and measures the effect of moving them on cache reuse and prefill.
Protocol Observatory: Side-by-side requests compare Ollama's native, OpenAI-compatible, and Anthropic-compatible protocols, including streaming metrics and response-envelope differences.
LLMxRay Benefits:
Find Slowdowns: Seeing model loading, prompt processing, generation, and tool time separately helps narrow down where an agent run is spending time.
Catch Prompt Loss: Visible cache breaks and cut prompt sections make it easier to spot when a model receives less context than the client intended.
Protect Shared Capacity: Comparing client settings and their effects helps teams identify requests that make a shared Ollama instance slower for others.
Tune Cache Reuse: Measured comparisons help assess whether changing prompt structure can reduce repeated processing without changing the prompt's words.
Use Cases:
Agent Run Debugging: Inspect a multi-step agent session to see each model call, its requested tools, and the results returned between calls.
Editor Request Review: Check what a coding editor sends to Ollama, including prompt size, cache reuse, and whether long instructions were cut.
Shared GPU Troubleshooting: Investigate why a local model server is slow by comparing client settings, queued requests, and loaded model memory.
Prompt Cache Tuning: Test whether moving changing values, such as timestamps, later in a prompt helps preserve cached content across turns.