What is Kiln?
Kiln is a workbench for building, evaluating, and improving AI systems, aimed at engineers and teams working with agents, retrieval-augmented generation, and language models. Its desktop app brings experiments, datasets, evaluations, and optimization into one workspace, while a Python library lets teams deploy Kiln tasks in production. Individuals can use the app and library for free, with optional Kiln Pro features for AI-assisted development and team plans.
Kiln Features:
Evaluation Datasets: Turn requirements into evaluations, generate synthetic examples and LLM judges, and keep datasets versioned with a repository.
RAG Builder: Add documents for extraction, chunking, embedding, indexing, and search, then compare configurations with automatically generated retrieval evaluations.
Agent Composition: Combine reusable capabilities, tools, and sub-agents, with support for handing off work and delegating tasks.
System Optimization: Run experiments to tune prompts, tools, skills, and models, and distill results into smaller models.
On-Demand Skills: Give agents reusable instructions that load when needed, with nested reference files and version tracking for each skill.
Team Workbench: Use the desktop app for project work, gather reviews and feedback from teammates, and deploy tasks with the Python library.
Kiln Benefits:
Measured Changes: Evaluation results make it easier to see whether a prompt, model, or retrieval adjustment improves quality rather than relying on intuition.
Less Rework: Reusing extraction and embedding work lets teams test alternative search strategies without starting the whole document pipeline again.
Focused Context: Loading skill instructions only when relevant leaves more of an agent’s context available for the task at hand.
Shared Quality Work: Teammates can contribute feedback, ratings, and data generation without needing to code, broadening who can help improve results.
Data Control: Keeping datasets and evaluations on a local machine and syncing through a repository gives teams control over where their project data lives.
Use Cases:
RAG Prototyping: Build a document-search prototype from PDFs, images, video, or audio and compare retrieval setups against generated question-and-answer evaluations.
Agent Quality Checks: Create evaluations for an agent and use them to check how changes to its prompts, tools, skills, or model affect its performance.
Procedure-Based Agents: Give an agent access to internal guidelines, standards, or procedures as skills that it can load selectively during a task.
Model Upgrade Testing: Compare a current model with a replacement by running experiments and scoring the results against the same evaluations.
Team Data Review: Invite product and engineering teammates to review outputs, provide ratings, and help generate or label evaluation data.